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Preface 


In  this  project,  “Evaluation  of  U.S.  Army  Asymmetric  Warfare  Adap¬ 
tive  Leader  Program  (AWALP),”  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC)  asked  RAND  Arroyo  Center  to  systemati¬ 
cally  evaluate  the  effectiveness  of  AWALP  and  to  design  a  set  of  instru¬ 
ments,  tools,  and  protocols  to  foster  ongoing  assessment  and  improve¬ 
ment  of  AWALP  and  other  courses  or  events  that  include  adaptability 
training. 

This  report  describes  the  study  design  and  results  based  on 
the  collection  of  data  from  three  AWALP  classes.  Using  a  pretest- 
posttest  study  design,  the  evaluation  addresses  multiple  outcomes, 
including  improvement  in  attitudes  toward  adaptability,  cognitive 
learning,  and  adaptability  behaviors;  reactions  to  the  course;  and  appli¬ 
cation  of  adaptability  principles  once  graduates  return  to  their  units.  The 
report  will  be  of  interest  to  those  seeking  to  measure  adaptability  and 
other  intangible  skills  in  institutional  and  unit  training  environments. 

This  research  was  sponsored  by  TRADOC  and  was  conducted 
within  the  RAND  Arroyo  Center’s  Manpower  and  Training  Program. 
RAND  Arroyo  Center,  part  of  the  RAND  Corporation,  is  a  feder¬ 
ally  funded  research  and  development  center  sponsored  by  the  United 
States  Army. 

The  Project  Unique  Identification  Code  (PUIC)  for  the  project 
that  produced  this  document  is  RANI 26497. 

Correspondence  regarding  this  report  should  be  addressed  to 
Susan  Straus  (sgstraus@rand.org)  or  Michael  Shanley  (mikes@rand. 
org).  Lor  more  information  on  the  RAND  Arroyo  Center,  contact 
the  Director  of  Operations,  Marcy  Agmon  (telephone  310-393-0411, 
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extension  6419;  FAX  310-451-6952;  e-mail  Marcy_Agmon@rand. 
org);  or  visit  Arroyo’s  website  at  http://www.rand.org/ard/. 
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Summary 


Introduction 

The  type  of  operations  the  U.S.  Army  has  confronted  in  the  post- 
September  11,  2001,  security  environment  has  mandated  increased 
adaptability  in  the  service.  Operational  success  requires  leaders  and 
organizations  that  can  rapidly  recognize  changes  in  the  environment, 
identify  critical  elements  in  unfamiliar  situations  with  less-than-perfect 
information,  and  facilitate  timely  action  to  meet  new  requirements — 
all  while  under  considerable  stress.  The  need  to  develop  adaptable  lead¬ 
ers  is  articulated  in  key  policy  documents,  including  the  U.S.  Army 
Learning  Concept  for  2015  and  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC)  strategic  plan  for  2013-2020,  and  it  is  an 
underlying  principle  in  the  Army’s  shift  from  a  doctrine  of  “command 
and  control”  to  “mission  command”  (TRADOC,  2011;  TRADOC, 
undated).  Joint  Chief  of  Staff  Chair  GEN  Martin  Dempsey,  in  his 
former  role  as  Commanding  General,  TRADOC,  stressed  the  need 
for  leaders  who  value  a  bottom-up  over  a  top-down  approach  and 
emphasized  the  importance  of  developing  teams  that  can  anticipate 
and  manage  transitions. 

As  part  of  the  Army’s  response  to  the  need  for  an  adaptive  force, 
the  Army’s  Asymmetric  Warfare  Group  (AWG)  implemented  the 
Asymmetric  Warfare  Adaptive  Leader  Program  (AWALP),  a  course 
designed  to  enhance  adaptability  in  leaders  and  to  promote  innovative 
solutions  in  the  training  for  and  conduct  of  unified  land  operations. 
The  primary  goals  of  the  course  are  to  enhance  adaptability  in  leaders, 
increase  understanding  of  the  need  for  adaptable  soldiers  and  enablers 
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of  adaptive  behavior,  and  develop  leader  ability  to  design  training  that 
fosters  adaptability  and  its  enablers. 

This  research  documented  here  evaluated  the  effectiveness  of 
AWALP  and  had  three  main  objectives.  The  first  was  to  provide  a  sys¬ 
tematic  evaluation  of  AWALP,  addressing  multiple  individual  and  team 
outcomes  and  identifying  potential  areas  for  improvement  in  AWALP 
curriculum  or  delivery.  A  related  aim  was  to  identify  facilitators  and 
challenges  to  applying  adaptability  principles  on  the  job  to  identify 
possible  changes  to  the  course  or  interventions  to  support  graduates  in 
implementing  adaptability  knowledge  and  skills. 

The  second  objective  was  to  provide  a  set  of  instruments,  tools,  and 
protocols  that  can  be  used  to  measure  adaptability  and  to  foster  ongo¬ 
ing  assessment  and  improvement  in  AWALP.  Guided  by  a  theoretical 
model,  we  designed  and  pilot  tested  a  number  of  different  measures  of 
training  outcomes  to  determine  which  provided  the  most  useful  data 
and  provided  a  set  of  final  materials  to  AWG  based  on  these  results. 

A  third  objective  was  to  provide  materials  suitable  not  only  for 
evaluating  AWALP  but  also  for  evaluating  other  courses  or  events 
that  include  adaptability  training  or  training  that  addresses  intangi¬ 
ble  skills  more  broadly  (e.g.,  many  aspects  of  teamwork  and  leadership 
skills).  Thus,  study  results  are  intended  to  support  not  only  AWG  but 
TRADOG  more  generally. 


AWALP  Structure,  Content,  and  Delivery 

AWALP  is  a  ten-day  course,  offered  four  times  a  year  at  Fort  A.P. 
Hill,  with  30  to  35  students  per  class.  Each  class  is  taught  by  a  team 
of  instructors  (who  are  called  guides),  with  a  student-to-guide  ratio 
of  approximately  8:1.  The  course  design  is  based  on  a  taxonomy  of 
adaptive  performance  (Pulakos  et  ak,  2000)  that  identified  eight 
dimensions  of  adaptable  performance,  such  as  solving  problems  cre¬ 
atively,  dealing  with  changing  or  ambiguous  situations,  interpersonal 
adaptability,  and  cultural  adaptability.  The  course  consists  of  about 
25-percent  classroom  instruction  and  75-percent  practical  exercises  or 
scenarios  requiring  adaptable  performance. 
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While  Army  training  frequently  focuses  on  standardized  pro¬ 
cedures  for  accomplishing  tasks,  AWALP  uses  an  outcomes-hased 
approach,  focusing  more  on  the  results  the  commander  intends  to 
achieve.  Although  how  tasks  are  executed  is  still  important,  achieving 
the  results  is  considered  to  he  more  important  than  the  actions  used 
to  attain  the  results  (as  long  as  the  actions  do  not  violate  the  com¬ 
mander’s  intent).  This  approach  encourages  trainees  to  take  initiative 
and  adjust  their  actions  to  adapt  to  the  situation,  which  require  inde¬ 
pendent  thinking  and  problem  solving.  After  a  training  event,  guides 
facilitate  an  after- action  review  (AAR)  that  promotes  self-discovery  of 
lessons  learned.  These  training  processes  promote  accountability  and 
foster  an  understanding  not  only  of  what  needs  to  be  done  but  why 
certain  actions  are  required — ^principles  that  are  aligned  clearly  with 
the  doctrine  of  mission  command. 


Research  Design 

We  addressed  individual,  team,  and  leadership  aspects  of  adaptabil¬ 
ity  while  addressing  significant  challenges  to  measuring  adaptability. 
These  challenges  include  the  need  to  measure  intangible  concepts,  the 
response  burden  that  a  comprehensive  analysis  can  impose,  and  a  lack 
of  established  measures  of  adaptability.  We  used  several  strategies  to 
address  these  challenges.  First,  we  used  multiple  approaches  to  mea¬ 
suring  adaptability,  including  piloting  new  ones,  to  assess  a  range  of 
adaptability  training  outcomes  and  to  offset  the  weaknesses  of  any  one 
method  or  measure  with  other  approaches.  Second,  for  most  outcome 
measures,  we  used  a  pretest-posttest  design  and  included  measures  of 
individual  differences  associated  with  adaptive  performance.  In  con¬ 
trast  to  posttest-only  designs  that  are  typically  used  in  training  evalu¬ 
ation,  this  approach  enabled  us  to  assess  the  impact  of  AWALP  train¬ 
ing.  Third,  we  balanced  the  comprehensiveness  of  the  approach  with 
response  burden  by  focusing  the  evaluation  on  the  factors  most  critical 
to  adaptability  and  to  the  levels  of  adaptability  addressed  in  AWALP. 

The  evaluation  included  assessment  of  outcomes  reflecting  differ¬ 
ent  goals  of  training.  These  included 
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•  reactions  to  the  course — satisfaction  with  course  content,  design, 
and  delivery 

•  attitudes  and  learning — how  training  results  in  changes  in  learn¬ 
ers 

•  transfer  performance  and  results — how  training  brings  about  pay¬ 
offs  to  the  organization. 


We  also  measured  individual  factors  that  influence  or  point  to 
reasons  for  training  outcomes.  Including  these  measures  can  improve 
the  ability  to  predict  or  explain  training  outcomes  and  can  enable 
designers  and  instructors  to  modify  aspects  of  the  course,  such  as 
prerequisites,  curriculum,  or  delivery,  to  improve  outcomes.  We  mea¬ 
sured  individual  characteristics  most  relevant  to  training  outcomes  in 
AWALP,  such  as  Big  Five  personality  traits,  general  cognitive  ability, 
learning  goal  orientation,  and  various  demographic  characteristics. 

The  evaluation  was  based  on  data  from  104  students  enrolled  in 
three  AWALP  classes  in  2013. 


Results 

Reaction  to  AWALP 

Students  were  extremely  satisfied  with  the  course  structure,  con¬ 
tent,  and  delivery.  AWALP  students  attribute  their  learning  largely  to 
the  content  of  the  course  and  the  training  methods,  which  differ  sub¬ 
stantially  from  typical  Army  training.  All  AWALP  students  reported 
that  they  would  recommend  the  course  to  others,  largely  because  of 
the  course  concepts,  the  training  approach,  and  improved  cognition. 
In  addition,  interviews  of  course  graduates  three  months  following 
AWALP  showed  that  reactions  remained  favorable  over  time.  Overall, 
we  found  that  students’  reactions  to  AWALP  were  much  more  favor¬ 
able  than  active-duty  leaders’  attitudes  about  institutional  training  in 
general,  as  reported  in  the  2013  Center  for  Army  Leadership  Annual 
Survey  of  Army  Leadership  (CASAL)  survey. 

Students  anticipated  that  leader  buy-in  and  command  cli¬ 
mate  would  pose  challenges  to  applying  adaptability  concepts  on 
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the  job.  Leader  buy-in  was  mentioned  most  often,  by  nearly  one-third 
of  the  students.  Command  climate  was  the  second  most  frequently 
anticipated  barrier,  reported  by  20  percent  of  the  students.  Despite 
these  expected  challenges,  almost  one-third  of  the  students  expressed  a 
sense  of  determination  to  overcome  these  challenges. 

Attitudes  Toward  Adaptability 

Pretraining  and  posttraining  surveys  measured  three  types  of  attitudes 
toward  the  dimensions  of  adaptive  performance:  self-efficacy,  interest 
in  being  adaptable,  and  the  frequency  with  which  students  engage  in 
relevant  behaviors  on  their  jobs  (pretest)  or  should  do  so  (posttest).  The 
surveys  asked  the  same  kinds  of  attitude  questions  for  behaviors  per¬ 
taining  to  putting  adaptability  into  leader  practice,  such  as  coaching 
others,  seeking  subordinate  input,  and  seeking  consensus  on  decisions. 
Results  show  improvement  in  almost  all  aspects  of  attitudes  toward 
adaptability.  After  the  course,  students  thought  they  were  better  at  being 
adaptable  in  all  dimensions  and  in  leader  practice  (see  Figure  S.l).  They 
also  reported  greater  interest  in  adaptability  than  before  the  course. 

Results  show  substantial  improvement  in  self-efficacy  and 
interest,  even  after  accounting  for  students’  individual  differences 
that  are  associated  with  adaptability.  Students’  self-efficacy  and 
interest  for  all  dimensions  of  adaptability  and  leader  practice  increased 
and  were  rated  highly  following  AWALP,  regardless  of  individual  char¬ 
acteristics,  such  as  openness  to  experience,  conscientiousness,  extraver¬ 
sion,  learning  and  performance  orientations,  motivation  for  training, 
and  time  in  service. 

Results  also  suggest  an  increase  in  the  perceived  need  for 
adaptability  and  adaptive  leader  behaviors  in  the  students’  cur¬ 
rent  jobs.  At  the  end  of  the  course,  students  saw  their  jobs  as  requir¬ 
ing  greater  adaptive  performance  with  respect  to  all  the  dimensions  of 
adaptability  and  the  majority  of  leader  behaviors  assessed.  Thus,  after 
the  course,  students  not  only  sensed  a  change  in  their  own  capa¬ 
bilities  and  interests  about  adaptability  but  appeared  to  see  their 
work  contexts  in  a  different  light. 
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Figure  S.1 

Self-Efficacy  Ratings  Pre-  and  Posttraining 
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Knowledge  of  Course  Concepts 

Students  demonstrated  increased  knowledge  of  AWALP  concepts. 

Average  scores  on  a  multiple- choice  knowledge  test  were  60  percent 
correct  pretraining  and  76  percent  correct  posttraining.  As  with  results 
for  self-efficacy  and  interest  in  adaptability,  students  showed  improved 
knowledge  regardless  of  other  characteristics  associated  with  test  per¬ 
formance,  including  general  cognitive  ability  and  level  of  education. 
These  results  indicate  that  AWALP  was  successful  at  fostering 
knowledge  gain  for  a  wide  range  of  students. 

Team  Performance 

We  developed  a  measure  to  assess  adaptability  at  the  team  level  for 
practical  exercises,  using  both  the  students  and  guides  as  raters  of  team 
adaptive  performance.  The  measure  required  raters  to  assess  whether 
the  dimensions  of  adaptive  performance  were  required  in  each  exercise 
and  to  rate  the  team’s  effectiveness  on  these  dimensions.  Participants 


Summary  xix 


were  asked  to  complete  the  rating  form  after  team  exercises  on  days  3, 
5,  and  7. 

The  similarity  between  students’  and  guides’  ratings  of  adapt¬ 
ability  requirements  varied  across  the  three  exercises.  For  example, 
students  failed  to  recognize  that  some  adaptability  dimensions  were  not 
relevant,  particularly  in  the  first  two  exercises.  However,  results  also 
indicate  that  students  were  starting  to  discriminate  between  dimen¬ 
sions  of  adaptability  that  were  and  were  not  required  by  the  third  exer¬ 
cise.  The  number  of  and  degree  of  discrepancies  between  students’  and 
guides’  ratings  also  decreased  over  time,  suggesting  that  students  were 
becoming  more  accurate  in  their  ratings  of  requirements. 

Figure  S.2  shows  effectiveness  ratings  aggregated  across  the  three 
exercises.  In  general,  both  students  and  guides  gave  favorable  rat¬ 
ings  of  team  performance.  Students’  average  scores  hovered  around  5, 
corresponding  to  an  “agree”  response,  and  most  of  the  guides’  ratings 
ranged  from  approximately  4  (i.e.,  agree  somewhat)  to  5  (Figure  S.2). 
However,  students  consistently  gave  higher  ratings  of  team  perfor¬ 
mance  than  guides  did.  As  might  be  expected  given  the  differences 

Figure  S.2 

Student  and  Guide  Ratings  of  Team  Effectiveness  Aggregated  Across 
Exercises 


— Students 
— Guides 


NOTE:  Phy  =  physical,  Int  =  interpersonal.  Am  =  dealing  with  ambiguity,  Cul  =  cultural, 
Str  =  decisionmaking  under  stress.  Plan  =  planning. 
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in  their  experience  levels,  students  also  generally  discriminated  less 
than  guides  did  in  rating  team  performance  across  the  different 
dimensions  of  adaptability.  As  shown  in  Figure  S.2,  students’  ratings 
of  team  performance  were  more  similar  across  the  items,  as  indicated 
hy  the  relatively  flatter  slope  for  their  ratings. 

At  the  same  time,  students  also  perceived  that  team  effectiveness 
was  increasing,  a  view  that  guides  did  not  necessarily  share.  In  the  first 
two  exercises,  students  rated  their  teams  higher  on  some  dimensions, 
and  guides  rated  teams  higher  on  others.  However,  in  the  third  exer¬ 
cise,  students  rated  their  teams  higher  than  guides  did  on  all  the  items. 
Thus,  students’  ratings  of  effectiveness  changed  over  time,  but  they 
appeared  to  become  more  confident  about  the  level  of  adaptability 
in  their  teams  as  the  course  progressed. 

Application  of  AWALP  Principles  on  the  Job 

We  conducted  telephone  interviews  of  students  and  their  supervi¬ 
sors  at  three  and  six  months  postgraduation  to  assess  the  longer-term 
impact  of  AWALP  on  adaptability  behavior  and  attitudes  after  students 
returned  to  their  units.  In  interviews  conducted  three  months  after 
AWALP,  graduates  reported  substantial  application  of  AWALP  prin¬ 
ciples  on  the  job,  especially  in  the  areas  of  coaching,  training,  delegat¬ 
ing  to  subordinates,  and  seeking  subordinate  input.  Moreover,  many 
graduates  reported  applying  outcomes-based  strategies  for  these  activi¬ 
ties,  giving  more  freedom  to  subordinates  to  address  tasks  and  chal¬ 
lenges.  For  most,  these  positive  effects  of  AWALP  were  sustained  six 
months  after  graduation.  Changes  were  less  common  in  other  areas, 
such  as  conducting  AARs  and  briefing  commanding  officers,  largely 
because  many  of  the  graduates  reported  no  opportunity  to  engage  in 
these  activities  in  their  current  roles.  While  AWALP  principles  were 
successfully  disseminated  to  graduates’  subordinates,  dissemination 
was  somewhat  less  successful  to  peers  and  commanders  and  through¬ 
out  the  unit. 

Graduates  also  remained  extremely  positive  about  AWALP 
three  months  after  the  course.  All  graduates  reported  that  they  would 
still  recommend  the  course  to  others,  and  few  recommended  course 
changes.  However,  graduates  also  saw  the  same  obstacles  to  applying 
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AWALP  principles  once  back  at  their  units,  with  command  climate  and 
entrenched  leadership  still  the  most  frequently  mentioned  challenges. 

Supervisors  were  also  positive  about  the  effect  of  AWALP  on  grad¬ 
uates’  behavior,  particularly  with  respect  to  observing  improvements 
in  training  planning  and  communications  with  the  commanding  offi¬ 
cers.  However,  results  should  be  interpreted  with  caution,  because  the 
sample  size  of  supervisors  was  extremely  small.  Difficulties  in  recruit¬ 
ing  supervisors  for  interviews  were  likely  partly  because  the  evaluation 
was  a  research  study  and  therefore  required  graduates’  consent  to  con¬ 
tact  their  supervisors.  We  recommend  several  strategies  for  recruiting 
supervisors  in  future  evaluation  efforts. 


Summary,  Conclusions,  and  Future  Directions 

The  results  of  this  evaluation  provide  evidence  of  AWALP ’s  success 
across  a  range  of  measures,  including  reactions  to  the  course,  changes 
in  learner  attitudes  and  cognitive  learning,  and  recognition  of  the  need 
for  adaptability  in  situations  encountered  on  the  job. 

Recommendations  to  Improve  AWALP 

Students  had  few  recommendations  for  improving  AWALP,  and  other 
study  results  showed  limited  need  for  modifications  to  the  course. 

One  recommendation  is  to  put  more  emphasis  on  anticipat¬ 
ing  and  responding  to  potential  challenges  to  implementing  AWALP 
principles  on  the  job.  At  the  end  of  the  course,  students  expected  that 
leader  buy-in  and  command  climate  would  pose  the  greatest  obstacles 
to  implementation,  and  interviews  with  graduates  three  and  six  months 
after  the  course  confirmed  that  these  factors  indeed  presented  obsta¬ 
cles.  Over  time,  we  expect  that  receptivity  to  adaptability  concepts  will 
improve  as  a  critical  mass  of  soldiers  participates  in  the  course  or  learns 
about  adaptability  in  other  Army  training.  In  the  meantime,  however, 
we  recommend  explicitly  discussing  in  AWALP  these  potential  obsta¬ 
cles  and  strategies  to  address  them. 

Second,  although  students  generally  were  satisfied  with  the  feed¬ 
back  guides  provided,  use  of  the  team  rating  instrument  within  an 
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AAR  could  further  enhance  feedback  and  might  foster  student  self- 
awareness  and  convergence  of  students’  and  guides’  ratings,  which 
remained  discrepant  on  day  7  of  the  course.  Student  ratings  could 
he  useful  in  several  ways;  for  example,  student  ratings  could  point  to 
topics  that  are  particularly  important  for  guides  to  prohe  in  the  AAR, 
and  asking  students  to  complete  the  team  rating  form  before  and  after 
the  AAR  (rather  than  just  before)  might  encourage  them  to  reflect 
more  deeply  on  their  performance.  Guides’  ratings  of  students’  indi¬ 
vidual  effectiveness  could  be  another  source  of  feedback.  We  discuss 
this  topic  below. 

Third,  while  AWALP  already  addresses  important  team  behav¬ 
iors,  model  development  since  the  inception  of  the  course  has  advanced 
theory  of  team  adaptability  and  related  outcomes.  The  curriculum 
could  be  enhanced  by  reinforcing  important  team-level  concepts  (e.g., 
mutual  monitoring  and  backup  behavior)  and  by  providing  instruction 
about  the  relationships  among  inputs  to  the  team,  throughputs,  and 
outcomes. 

Fourth,  variation  in  students’  responses  across  different  mea¬ 
sures  used  in  the  evaluation  and/or  discrepancies  between  students’ 
assessments  and  guides’  evaluations  suggest  that  instruction  could 
be  enhanced  for  some  adaptability  dimensions.  These  include  han¬ 
dling  ambiguous  situations,  generating  innovative  ideas,  and  cultural 
adaptability. 

Finally,  a  related  topic  for  AWG  consideration  is  whether  stu¬ 
dents’  posttraining  knowledge  test  scores  are  acceptable.  If  they  are  not 
satisfactory,  additional  reinforcement  of  course  concepts  in  classroom 
instruction  and  during  practical  exercises  may  be  needed.  Likewise,  if 
scores  on  ratings  of  team  performance  are  not  satisfactory,  additional 
strategies  may  be  needed  to  enable  students  to  demonstrate  improved 
adaptability  in  team  contexts. 

Recommended  Measures  and  Processes  for  Ongoing  Evaluation 

We  recommend  that  AWG  continue  to  administer  several  of  the  instru¬ 
ments  used  in  this  evaluation,  including  the  knowledge  test,  questions 
about  attitudes  toward  adaptability  and  leader  practice,  reactions  to 
the  course,  and  ratings  of  team  adaptive  performance.  We  propose  a 
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number  of  revisions  to  these  measures  to  reduce  the  response  burden 
and  to  better  capture  important  aspects  of  team  collaboration  and  per¬ 
formance,  such  as  planning,  mutual  monitoring,  and  backup  behav¬ 
ior.  We  also  recommend  administering  the  instruments  online  rather 
than  using  paper  and  pencils  to  improve  the  efficiency  and  accuracy  of 
data  collection,  allow  responses  to  be  scored  immediately,  and  allow 
the  collection  of  additional  information  about  the  quality  of  students’ 
responses. 

We  also  recommend  continuing  to  assess  how  graduates  apply 
AWALP  principles  on  the  job  through  interviews  of  course  gradu¬ 
ates.  As  another  indicator  of  success,  we  recommend  tracking  multiple 
enrollments  from  the  same  units  over  time.  In  addition,  a  number  of 
different  Army  divisions  have  asked  AWG  for  support  in  standing  up 
local  adaptability  training  modeled  after  AWALP.  These  requests  can 
serve  as  additional  indicators  of  AWALP  success. 

Recommendations  for  Future  Evaluation  of  AWALP 

Additional  methods  and  measures  can  contribute  to  a  more  compre¬ 
hensive  evaluation  of  AWALP.  A  theme  of  many  of  these  recommen¬ 
dations  is  the  need  to  obtain  additional  measures  of  adaptive  perfor¬ 
mance.  Some  suggestions  are  as  follows: 

•  Most  important  is  assessing  transfer  of  training  by  assessing  the 
association  of  individual  characteristics  and  outcomes  measured 
in  training  with  the  quality  of  subsequent  job  performance.  To 
accomplish  this,  we  propose  that  guides  rate  student  performance 
in  AWALP  and  that  graduates’  supervisors  provide  independent, 
quantifiable  ratings  of  subsequent  job  performance.  For  greater 
success  in  contacting  supervisors,  we  recommend  modifying  the 
recruiting  processes  used  in  the  current  study.  Specific  recom¬ 
mendations  include  obtaining  supervisor  contact  information  on 
student  enrollment  in  AWALP,  contacting  supervisors  after  three 
months  rather  than  after  a  longer  time  interval,  and  giving  pri¬ 
ority  for  future  AWALP  enrollments  to  supervisors  who  provide 
feedback  about  course  graduates. 
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•  Assessing  transfer  of  training  could  be  expanded  further  by  col¬ 
lecting  feedback  from  not  only  graduates’  supervisors  but  also 
from  graduates’  peers  and  subordinates.  Assessment  could  be 
strengthened  by  including  a  matched  sample  of  leaders  who  have 
not  attended  AWALP.  The  Army’s  Multi-Source  Assessment  and 
Feedback  Program,  a  360-degree  evaluation  process  for  officers, 
could  serve  as  a  model  for  this  effort. 

•  More-extensive  behavioral  observations  of  individuals  or  teams 
could  be  used  to  obtain  additional  measures  of  effectiveness  in 
AWALP  and  to  assess  improvement  in  performance  as  the  course 
progresses.  For  example,  guides  could  use  behavior  checklists  to 
rate  the  frequency  of  adaptability  behaviors  exhibited  in  paral¬ 
lel  practical  exercises  conducted  at  the  beginning  and  end  of  the 
course  (e.g.,  Roselle,  2013).  As  mentioned  above,  ratings  of  indi¬ 
viduals  would  also  provide  predictor  measures  for  studying  train¬ 
ing  transfer. 

Expanding  AWALP's  Approach  to  Adaptability  Training 

The  Army  should  consider  ways  to  leverage  AWG’s  approach  by  incor¬ 
porating  AWALP  principles  into  other  Army  training.  One  strategy  is 
to  increase  the  number  of  trainees  in  the  existing  course  or  modified 
versions  of  it.  For  example,  AWG  could  continue  to  support  standing 
up  AWALP  training  in  Army  divisions  by  training  local  trainers  or 
through  mobile  training  teams.  Another  way  to  disseminate  AWALP 
more  broadly  is  to  incorporate  adaptability  principles  into  existing  pro¬ 
fessional  military  education  courses  required  for  promotion,  such  as 
advanced  leader  courses,  senior  leader  courses,  the  Basic  Officer  Leader 
Gourse,  Gaptains’  Gareer  Gourse,  and  intermediate-level  education. 

TRADOG  can  also  support  mission  command  principles  by 
expanding  on  instruction  of  team  adaptability.  AWALP  provides 
a  starting  point  for  training  soldiers  to  work  in  and  lead  teams,  but 
there  are  many  additional  topics  that  foster  effective  teams  and  effec¬ 
tive  team  leaders.  Given  the  large  and  growing  literature  relevant  to 
team  adaptability,  we  recommend  developing  an  AWALP  follow-on 
course  focused  exclusively  on  these  topics.  Alternatively,  some  existing 
leader- development  institutional  courses  or  other  training  that  focuses 
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on  team  interaction  and  performance  might  also  provide  an  appropri¬ 
ate  context  for  this  training.  Examples  of  prospective  team-hased  train¬ 
ing  topics  include  shared  mental  models,  transactive  memory  systems, 
team  trust,  process  losses  in  group  tasks,  and  team  facilitation. 

To  support  expansion  of  AWG’s  approach  to  adaptability  train¬ 
ing,  we  recommend  that  AWG  create  a  training  support  package  for 
AWALP  with  a  program  of  instruction  and  supplementary  materials. 
While  a  conventional  Army  program  of  instruction  may  not  he  appro¬ 
priate  for  AWALP  in  light  of  its  teaching  approach,  a  set  of  organized 
materials  will  help  preserve  institutional  knowledge  related  to  train¬ 
ing  adaptability  and  will  support  dissemination  of  course  content  and 
instructional  methods  to  the  Army  at  large. 

Lessons  for  Adaptability  Evaluation  in  Other  Training  Contexts 

Lessons  learned  from  this  research  apply  to  measuring  adaptability  and 
other  intangible  concepts  not  only  in  AWALP  but  also  in  other  train¬ 
ing  contexts.  The  evaluation  demonstrates  the  benefits  of  using  mul¬ 
tiple  measures  and  methods  and  documents  how  such  measures  can 
be  developed  and  implemented  to  assess  intangible  training  outcomes. 
Evaluations  of  other  courses  that  involve  adaptability  training  can 
make  use  of  most  of  the  reactions  measures  implemented  in  this  study 
“as  is”  or  with  minor  modifications.  Pre-post  knowledge  tests,  attitudi- 
nal  measures,  and  postgraduate  interviews  can  be  used  with  appropri¬ 
ate  revisions  for  course  content.  Instructor  and  student  ratings  of  team 
performance  may  be  broadly  applicable  to  Army  training.  Gontinued 
use  of  these  instruments,  including  the  modifications  suggested  above 
and  described  in  more  detail  in  this  report,  can  provide  data  to  validate 
the  measures  and  support  TRADOG  in  evaluating  a  wide  range  of 
training  efforts  that  emphasize  21st-century  soldier  skills. 


Conclusion 

The  shift  in  Army  doctrine  from  command  and  control  to  mission  com¬ 
mand  calls  for  profound  changes  in  leader  and  team  conduct.  These 
changes  in  philosophy  require  a  concomitant  transformation  in  train- 
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ing.  AWG’s  successful  development  and  implementation  of  AWALP 
exemplifies  mission  command  principles  in  terms  of  both  the  content 
of  the  course  and  the  manner  in  which  it  is  taught.  AWALP,  supported 
by  systematic  course  evaluation,  provides  a  promising  approach  for  the 
Army  as  it  seeks  to  further  develop  adaptable  leaders  and  teams. 
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AAR 

after- action  review 

ADP 

Army  Doctrine  Publication 

Am 

dealing  with  ambiguity 

ATOM 

Anti-Air  Teamwork  Observation  Measure 

AWALP 

Asymmetric  Warfare  Adaptive  Leader  Program 

AWG 

Asymmetric  Warfare  Group 

CASAL 

Center  for  Army  Leadership  Annual  Survey  of  Army 
Leadership 

Cre 

creative  thinking 

Cul 

cultural 

FFI 

Live-Lactor  Inventory 

ICC 

intraclass  correlation 

Int 

interpersonal 

NCO 

noncommissioned  officer 

OBT&E 

outcomes-based  training  and  education 

Phy 

physical 

Plan 

planning 
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PME  professional  military  education 

POI  program  of  instruction 

PT  physical  training 

Str  decisionmaking  under  stress 

TRADOC  Training  and  Doctrine  Command 


CHAPTER  ONE 


Introduction 


Background 

The  Need  for  Adaptability 

The  type  of  operations  the  U.S.  Army  has  confronted  in  the  post- 
September  11,  2001,  security  environment  has  shown  the  need  for 
increased  adaptability  in  the  service.  For  example,  as  Burke,  Pierce, 
and  Salas,  2006,  p.  ix,  notes: 

The  US  military  is  facing  an  increasingly  complex  geopolitical 
environment  that  also  demands  adaptability  in  order  to  be  effec¬ 
tive.  For  example,  within  the  military,  individuals  are  having  to 
adapt  to  asymmetric  threats,  increased  joint  operations,  and  net¬ 
work  capabilities.  Teams  are  having  to  adapt  to  a  wide  variety 
of  environmental  and  team  composition  factors  (e.g.,  warfight¬ 
ing,  peacekeeping,  teams  comprised  of  coalition  partners  of  mul¬ 
tiple  nationalities),  and  at  the  organizational  level  the  military  is 
having  to  adapt  to  changes  in  the  geopolitical  environment. 

The  Army  must  develop  soldiers  prepared  for  operations  in  an  envi¬ 
ronment  that  is  complex,  ambiguous,  and  highly  uncertain  and  in 
one  that  demands  action  within  shorter  time  frames  (Zaccaro  et  ah, 
2009).  Operational  success  will  require  leaders  and  organizations  that 
can  rapidly  adapt  by  recognizing  changes  in  the  environment,  identify¬ 
ing  critical  elements  of  the  new  situation  with  less-than-perfect  infor¬ 
mation,  and  facilitating  timely  action  to  meet  new  requirements — all 
while  under  considerable  stress. 
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Senior  Army  leaders  have  emphasized  the  importance  of  devel¬ 
oping  a  more-adaptive  force  and  have  identihed  “adaptability  and  ini¬ 
tiative”  as  key  21st-century  soldier  competencies  (U.S.  Training  and 
Doctrine  Command  [TRADOC],  2011).  According  to  Army  Chief 
of  Staff  GEN  Ray  Odierno,  developing  adaptive  leaders  is  a  top  pri¬ 
ority  for  the  Army  (38th  Army  Chief  of  Staff,  2013).  Joint  Chief  of 
Staff  Chair  GEN  Martin  Dempsey,  in  his  former  role  as  Commanding 
General,  TRADOC,  emphasized  the  need  to  seek  and  embrace  adapt¬ 
ability  to  confront  hybrid  threats.  In  describing  a  shift  from  a  doctrine 
of  “command  and  control”  to  “mission  command,”  General  Dempsey 
also  stressed  the  need  for  teams  that  can  anticipate  and  manage  tran¬ 
sitions  and  leaders  who  value  a  bottom-up  over  a  top-down  approach 
and  who  “pass  resources  and  responsibility  ‘to  the  edge’”  (Dempsey, 
2011,  p.  44).  The  exercise  of  mission  command,  as  described  in  Army 
Doctrine  Publication  (ADP)  6-0,  2012,  p.  1,  calls  for  commanders  to 
“empower  agile  and  adaptive  leaders”  guided  by  principles  that  include 
building  cohesive  teams  through  mutual  trust,  creating  shared  under¬ 
standing,  and  exercising  disciplined  initiative.  Eikewise,  in  TRADOC’s 
strategic  plan  for  2013-2020,  Commanding  General  of  TRADOC, 
GEN  Robert  Cone,  emphasized  the  need  to  train  leaders  to  be  adapt¬ 
able  (TRADOC,  undated). 

As  part  of  the  Army’s  response  to  the  need  for  an  adaptive  force, 
the  Army’s  Asymmetric  Warfare  Group  (AWG)  implemented  the 
Asymmetric  Warfare  Adaptive  header  Program  (AWAEP)  in  2011, 
a  course  designed  to  enhance  adaptability  in  leaders  and  to  promote 
innovative  solutions  in  training  and  unified  land  operations.  AWG, 
in  turn,  asked  RAND  Arroyo  Center  to  conduct  an  evaluation  of  the 
effectiveness  of  AWAEP.  This  document  represents  the  final  report  on 
that  evaluation. 

Definition  and  Dimensions  of  Adaptability 

AWG’s  definition  of  adaptability,  which  we  also  use,  comes  from 
Army  Field  Manual  6-22,  2006.  The  shorter  version  of  that  definition, 
which  is  used  in  AWAEP  training  materials,  defines  adaptability  as  “an 
effective  change  in  behavior  in  response  to  an  altered  situation.”  An 
expanded  version,  also  cited  in  AWG’s  leaders  guide,  emphasizes  the 
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individual’s  role  in  bringing  about  the  needed  change:  “Adaptability  is 
an  individual’s  ability  to  recognize  change  in  the  environment,  identify 
the  critical  elements  of  the  new  situation,  and  trigger  changes  accord¬ 
ingly  to  meet  new  requirements.” 

The  conceptual  basis  for  AWALP  design  comes  from  the  Pula- 
kos  et  al.,  2000,  model  of  adaptive  performance.  Pulakos  et  al.,  2000, 
presents  and  tests  a  taxonomy  positing  eight  dimensions  of  adaptive 
performance  (Figure  1.1).'  (See  Chapter  Two  for  a  description  of  how 
the  taxonomy  was  developed  and  tested.)  AWG  staff  adopted  the 
Pulakos  et  al.  taxonomy  for  several  reasons:  It  is  well-documented,  and 
its  focus  on  observable  behaviors  of  the  adaptive  performance  dimen¬ 
sions  supported  the  design  of  course  activities;  the  research  used  to 
establish  the  dimensions  included  a  large  number  of  military  respon¬ 
dents,  which  contributed  to  buy-in  among  Army  stakeholders;  and 
U.S.  Army  Research  Institute  research  documented  application  of 


Figure  1.1 

The  Eight  Dimensions  of  Adaptability 
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^  Some  researchers  distinguish  between  the  terms  adaptation  and  adaptability,  with  the 
former  referring  to  performance  and  the  latter  to  an  individual  characteristic  (e.g.,  see 
Baard,  Rench,  and  Kozlowski,  2014).  However,  AWG  and  this  report  use  these  terms 
interchangeably. 
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the  model  in  an  adaptive  thinking  and  leadership  course  for  Special 
Forces  officer  candidates  (White  et  ah,  2005)  and  found  additional 
support  for  the  model  (Tucker  and  Gunther,  2009;  Zaccaro  et  ah, 
2009).  Although  there  is  some  evidence  that  there  may  he  a  simpler 
structure  with  fewer  than  eight  dimensions  of  adaptive  performance 
(e.g.,  Griffin  and  Hesketh,  2003;  Pulakos  et  ah,  2002),  Baard,  Rench, 
and  Kozlowski,  2014,  recommend  using  the  Pulakos  et  ah,  2000,  tax¬ 
onomy  as  a  starting  point  for  research  in  light  of  its  rigorous  theoretical 
and  empirical  foundations,  coupled  with  a  lack  of  theory  and  system¬ 
atic  research  in  alternative  formulations  of  adaptive  performance  (and 
indeed,  the  Pulakos  et  ah,  2000,  taxonomy  has  been  very  influential  in 
other  theoretical  perspectives,  e.g.,  Ployhart  and  Bliese,  2006).^ 

In  AWALP,  four  of  the  eight  adaptive  performance  dimensions 
(shown  in  the  middle  of  Figure  1.1)  are  considered  to  he  core,  two 
dimensions  are  considered  to  he  enabling,  and  two  are  considered  to  be 
supporting.^  The  dimensions  are  defined  in  Table  1.1.^ 

AWALP  Structure  and  Content 

AWALP  is  a  ten-day  course  for  noncommissioned  officers  (NGOs)  and 
junior  officers  primarily  in  the  active  component.  It  began  in  2011 
and  is  currently  taught  four  times  a  year  in  residence  at  Fort  A.P.  Hill 
to  about  30  to  35  students  per  class.  Each  class  is  taught  by  a  team  of 
instructors,  called  guides,  which  reflects  their  primary  role  as  facilita¬ 
tors  rather  than  teachers.  The  guides  initially  selected  for  the  course 
were  highly  experienced,  retired  special  operations  forces  operators 
with  a  demonstrated  proficiency  in  teaching  marksmanship  and  tacti¬ 
cal  problem  solving  and  who  supported  the  guiding  principles  of  the 


^  For  example,  Entin  and  Serfaty,  1999,  found  a  three-factor  solution  of  proactive  behav¬ 
iors,  such  as  thinking  creatively;  reactive  behaviors,  such  as  interpersonal  and  cultural  adapt¬ 
ability;  and  tolerant  behaviors,  such  as  handling  stress.  The  overlap  with  the  Pulakos  et  ah, 
2000,  taxonomy  is  evident. 

^  AWG  further  organized  the  dimensions  into  three  larger  categories:  physical  (physical 
adaptability),  interpersonal  (interpersonal  adaptability  and  cultural  adaptability),  and  cogni¬ 
tive  (the  four  core  dimensions  in  Figure  1.1,  plus  “learning  tasks,  technologies,  procedures.”) 

^  The  definitions  are  based  on  those  in  Asymmetric  Warfare  Adaptive  Leader  Program, 
Leader’s  Guide  for  Enhancing  Adaptability,  December  2011,  pp.  4—5. 
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Table  1.1 

Definitions  of  Adaptability  Dimensions 
Dimension  Definition 


Thinking  creatively  Developing  innovative  solutions  to  problems  by  entertaining 
a  wide  range  of  options,  developing  innovative  methods  of 
achieving  outcomes  with  limited  resources,  and  integrating 
seemingly  unconnected  information  into  logical  patterns  that 
increase  situational  awareness 


Dealing  with 
changing  or 
ambiguous 
situations 

Learning 
work  tasks, 
technologies,  and 
procedures 


Interpersonal 

adaptability 


Cultural 

adaptability 


Decisionmaking 
under  stress 


Handling 
emergencies  and 
crises 


Physical 

adaptability 


Using  critical  thinking  to  define  the  nature  of  a  problem; 
taking  effective  action  when  needed,  even  without  all  desired 
information;  and  responding  to  changing  environments  with 
cognitive  flexibility 

Taking  action  to  improve  work  performance  through  a  process 
of  self-reflection 

By  taking  action,  learning  new  skills  or  gaining  information 
in  response  to  an  altered  situation  or  preparing  for  future 
possibilities 

Participating  and  working  well  with  others  as  a  team  member 
and  building  positive,  effective  relationships,  including  listening 
to  and  understanding  others 
Learning  how  to  effectively  lead  a  discussion 

Communicating  effectively  across  a  diversity  of  cultures, 
regardless  of  differences 

Being  willing  to  adjust  behavior  to  comply  with  or  show  respect 
for  others'  values  and  customs 

Remaining  resilient  and  taking  appropriate  actions  when 
confronted  by  various  stressors 
Being  a  calming  influence  on  others 

Reacting  effectively  in  life-threatening  situations  (e.g.,  staying 
clear  and  focused  and  making  swift  decisions) 

Having  ability  to  apply  emergency  techniques  and  skills  when 
accidents  or  injuries  occur 

Having  or  training  to  obtain  the  weight,  strength,  and 
endurance  necessary  for  your  job  and  to  go  beyond  it 
Adjusting  oneself  to  complete  demanding  tasks  and  to  operate 
in  varied  environments  and  under  austere  conditions 


NOTE:  AWG's  terms  for  some  of  the  dimensions  are  slightly  different  from  those  of 
Pulakos  et  al.,  2000.  Pulakos  et  al.  used  "Solving  problems  creatively"  rather  than 
"Thinking  creatively"  and  used  "Dealing  with  uncertain  and  unpredictable  work 
situations"  rather  than  "Dealing  with  changing  or  ambiguous  situations."  These 
variations  in  terminology  do  not  affect  the  substantive  focus  of  the  dimensions. 
However,  while  Pulakos  et  al.,  2000,  identified  a  dimension  entitled  "Handling  work 
stress,"  AWALP  focuses  specifically  on  "decisionmaking  under  stress." 
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course.  Later  guides  were  selected  to  have  the  same  or  similar  skills  and 
attitudes.  The  student-to-guide  ratio  is  approximately  8:1.  The  course 
is  unlike  most  Army  institutional  training  in  that  it  does  not  have  a 
formal  program  of  instruction  (POI)  with  enabling  or  terminal  learn¬ 
ing  objectives.  The  primary  goals  of  the  course  are  to  enhance  adapt¬ 
ability  in  leaders,  increase  understanding  of  the  need  for  adaptable  sol¬ 
diers  and  enablers  of  adaptive  behavior, ^  and  develop  leader  ability  to 
design  training  that  fosters  adaptability  and  its  enablers. 

The  course  presents  a  variety  of  learning  experiences,  consisting  of 
about  75  percent  hands-on  activities  and  25  percent  classroom  instruc¬ 
tion.  The  course  seeks  to  fundamentally  transform  students’  thinking 
about  what  it  means  to  be  a  leader  and  trainer  and  how  to  get  the  most 
out  of  the  teams  they  lead.  The  first  phase  of  the  course  (approximately 
the  first  five  days)  uses  classroom  activities  and  focused  practical  field 
exercises  to  develop  individual  and  team  adaptability.  This  phase  is 
devoted  to  ensuring  that  participants  understand  the  dimensions  of 
adaptive  performance,  how  adaptability  fosters  individual  and  team 
performance,  and  how  that  performance  leads  to  the  agility  and  opera¬ 
tional  adaptability  needed  for  current  operations.  The  second  phase 
presents  more-complex  problems  reflecting  contemporary  operational 
challenges.  This  phase  is  intended  to  help  students  learn  to  respond  to 
asymmetric  threats  in  different  operational  environments.  The  second 
ph  ase  also  places  special  emphasis  on  developing  leaders’  abilities  to 
design  unit  training  that  integrates  adaptability  concepts. 

The  course  does  not  cover  each  of  the  dimensions  of  adaptive  per¬ 
formance  shown  in  Table  1.1  to  an  equal  degree.  Creative  thinking; 
dealing  with  ambiguity;  learning  tasks,  technologies,  and  procedures; 
interpersonal  adaptability;  and  handling  stress  are  embedded  into 
nearly  all  course  events.  Cultural  adaptability  is  represented  somewhat 
less.  The  course  puts  little  emphasis  on  physical  adaptability  because 
the  terrain  at  the  training  site  is  not  conducive  to  conducting  practical 


^  While  two  of  the  dimensions  are  considered  enabling,  the  term  enabler  here  refers  to  other 
factors  that  foster  adaptive  behavior.  These  include  opportunities  for  trainees  to  test  their 
confidence,  practice  decisionmaking,  practice  innovative  problem  solving,  and  demonstrate 
initiative,  all  with  an  awareness  of  accountability. 
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exercises  that  test  this  dimension.  Likewise,  the  course  does  not  address 
responding  to  crisis  situations  because  it  lacks  appropriate  training 
facilities  to  create  high-fidelity  crisis  scenarios. 

The  curriculum  is  highly  experiential.  Each  day,  students  com¬ 
plete  one  or  more  practical  exercises  that  are  designed  to  tap  a  subset 
of  the  adaptive  performance  dimensions  (see  Appendix  A  for  an  exem¬ 
plar  exercise).  Some  of  the  exercises  create  realistic,  simulated  military 
scenarios  (as  described  in  Chapter  Four),  while  others  consist  of  more- 
general  problem-solving  activities  (e.g.,  puzzles,  tower  building). 
Instructional  practices  largely  reflect  constructivist  educational  prin¬ 
ciples,  many  of  which  are  also  consistent  with  recommended  strate¬ 
gies  for  training  for  team  adaptation  (see  Burke,  Stagl,  et  ah,  2006). 
Common  constructivist  strategies  include  (Gagne  et  ah,  2005): 

•  working  in  collaborative  groups 

•  presenting  novel  problems  to  solve  in  realistic  settings  (called  situ¬ 
ated  learning  or  problem-based  learning',  e.g.,  Anderson,  Reder  and 
Simon,  1996;  Lave  and  Wenger,  1991;  Savery  and  Duffy,  1995). 

•  challenging  students  with  tasks  just  beyond  their  current  skill 
level  to  motivate  them  and  build  their  confidence 

•  providing  opportunities  for  practice  in  similar  situations 

•  providing  guidance  from  instructors  prepared  to  intervene  as 
needed  (called  guided  discovery) 

•  encouraging  students  to  reflect  on  the  learning  process  and  their 
skill  acquisition. 

In  addition,  while  much  of  Army  training  frequently  focuses  on 
learning  standardized  procedures  to  accomplish  tasks,  AWALP  uses 
an  outcomes-based  approach,  focusing  more  on  the  results  the  com¬ 
mander  hopes  to  achieve.'"  Although  how  tasks  are  executed  is  still 
important,  achieving  the  results  is  considered  to  be  more  important 


This  approach  is  referred  to  as  outcomes-based  training  and  education  (OBT&E).  OBT&E 
concepts  were  developed  in  the  Army  (see  Elaskins,  2010)  and  are  similar  to  ideas  of 
outcomes-based  education  as  discussed  by  William  Spady  (Cornell-d’Echert,  personal  com¬ 
munication,  November  12,  2013).  While  OBT&E  concepts  are  similar  to  some  established 
theories  in  organizational  behavior,  such  as  goal  setting,  we  have  not  been  able  to  find  empir¬ 
ical  research  investigating  OBT&E. 
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than  the  actions  used  to  attain  the  outcomes  (as  long  as  the  actions  do 
not  violate  the  commander’s  intent).  Consistent  with  constructivism, 
this  gives  trainees  the  freedom  to  take  initiative,  adjust  their  actions, 
and  adapt  to  the  situation,  which  require  them  to  engage  in  indepen¬ 
dent  thinking  and  problem  solving.  After  a  training  event,  the  instruc¬ 
tor  facilitates  an  after- action  review  (AAR)  that  promotes  self-discovery 
of  lessons  learned.  These  training  processes  promote  accountability  and 
an  understanding  not  only  of  what  needs  to  be  done  but  why  certain 
actions  are  required,  which,  in  turn,  is  thought  to  empower  trainees 
and  foster  their  confidence.  These  instructional  strategies  are  aligned 
clearly  with  the  principles  of  mission  command.  According  to  ADP 
6-0,  2012,  to  support  decisionmaking  at  the  point  of  action,  com¬ 
manders  must 

concentrate  on  the  objectives  of  an  operation,  not  how  to  achieve 
it.  Commanders  provide  subordinates  with  their  intent,  the  pur¬ 
pose  of  the  operation,  the  key  tasks,  the  desired  end  state,  and 
resources.  Subordinates  then  exercise  disciplined  initiative  to 
respond  to  unanticipated  problems. 


Objectives 

In  response  to  AWG’s  request,  we  designed  and  conducted  an  evalu¬ 
ation  of  AWALP  to  achieve  three  main  objectives.^  The  first  was  to 
systematically  evaluate  the  effectiveness  of  AWALP,  addressing  mul¬ 
tiple  individual  and  team  outcomes.  The  evaluation  focused  on  four 
primary  customers:  AWALP  program  designers,  guides  (i.e.,  instruc¬ 
tors),  course  graduates,  and  TRADOC.  One  evaluation  focus  was  on 
identifying  potential  areas  for  improvement  in  AWALP  itself,  either  in 
the  curriculum  or  in  how  it  is  taught,  by  measuring  training  outcomes 


^  At  the  time  that  we  began  our  research,  AWG  was  engaged  in  a  study  examining  changes 
in  attitudes  toward  adaptability  and  adaptable  performance  from  the  beginning  to  the  end 
of  the  course,  along  with  interviews  of  graduates  to  assess  their  views  of  the  course  and  how 
it  had  affected  their  jobs  (Roselle,  2013).  Because  this  study  was  in  progress,  results  were  not 
available  to  inform  the  design  of  our  evaluation;  however,  we  have  integrated  the  findings 
into  our  report  where  appropriate. 
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during  and  immediately  following  the  course.  A  related  aim  was  to 
identify  facilitators  and  challenges  to  applying  adaptability  principles 
on  the  joh,  with  the  goal  of  using  the  results  to  suggest  additional 
changes  in  the  course  content  or  interventions  to  support  graduates  in 
implementing  adaptability  concepts  once  they  return  to  their  jobs. 

The  second  study  objective  was  to  provide  a  set  of  instruments, 
tools,  and  protocols  that  could  be  used  to  measure  adaptive  perfor¬ 
mance  and  to  foster  ongoing  assessment  and  improvement  in  AWALP. 
Guided  by  a  theoretical  model  of  training  evaluation  and  effective¬ 
ness,  we  designed  and  pilot  tested  a  number  of  different  measures  to 
determine  which  provided  the  most  useful  data.  Final  materials  differ 
somewhat  from  those  used  during  the  project  because  our  evaluation 
included  efforts  to  minimize  response  burden  and  to  simplify  adminis¬ 
tration  and  analyses  required  on  an  ongoing  basis. 

A  third  study  objective  was  to  provide  materials  suitable  not  only 
for  evaluating  AWALP  but  for  evaluating  other  courses  or  events  that 
include  adaptability  training  or  training  that  addresses  intangible 
skills  more  broadly  (e.g.,  many  aspects  of  teamwork  and  leadership 
skills).  Thus,  study  results  are  intended  to  support  not  only  AWG  but 
TRADOG  more  generally. 


Organization  of  This  Report 

Ghapter  Two  presents  a  brief  review  of  adaptability  from  the  research 
literature  and  describes  our  study  design,  which  is  driven  by  that  review. 

We  present  more-detailed  descriptions  of  the  methods  and  anal¬ 
yses  and  results  from  our  evaluation  in  the  three  subsequent  chap¬ 
ters.  Ghapter  Three  addresses  students’  reactions  and  attitudes,  and 
Ghapter  Four  addresses  student  learning.  Ghapter  Five  describes  grad¬ 
uates’  application  of  AWALP  principles  on  the  job  and  graduates’  atti¬ 
tudes  toward  AWALP  after  returning  to  their  units. 

Ghapter  Six  summarizes  our  hndings  and  conclusions  and  pres¬ 
ents  recommendations  for  revising  the  course,  for  ongoing  evaluation 
of  AWALP,  and  for  future  evaluation  of  AWALP  and  related  courses. 

Finally,  a  series  of  appendixes  provides  supplementary  information. 


CHAPTER  TWO 


General  Approach  and  Study  Design 


We  begin  with  an  overview  of  the  general  approach  that  guided  our 
study  design,  particularly  in  selecting  training  evaluation  and  effective¬ 
ness  measures.  This  includes  a  brief  review  of  findings  from  the  research 
literature  that  bear  on  the  content  of  the  measures  and  a  description 
of  general  challenges  in  measuring  adaptability.  We  then  discuss  our 
study  design  and  how  the  design  addresses  the  challenges. 


General  Approach 

Underlying  Model  of  Training  Evaluation  and  Effectiveness 

Our  evaluation  design  was  guided  by  a  model  of  training  evaluation 
and  effectiveness  adapted  from  Alvarez,  Salas,  and  Garofano,  2004, 
which  we  have  used  in  prior  research  (Straus,  Shanley,  Lytell,  et  ah, 
2013;  Straus,  Shanley,  Yeung,  et  ah,  2011;  see  the  last  report  for  a 
detailed  description  of  the  model).  The  model  identifies  outcome  mea¬ 
sures  to  evaluate  training  and  factors  that  may  influence  or  explain 
these  outcomes.  Outcome  measures  indicate  whether  a  course  meets 
its  intended  goals  (e.g.,  whether  students  learn  the  material  or  apply 
trained  skills  on  the  job).  Alvarez,  Salas,  and  Garofano,  2004,  identifies 
five  types  of  outcome  measures  based  on  the  four-level  classification 
system  (reactions,  learning,  behavior  or  on-the-job  performance,  and 
results)  in  Kirkpatrick,  1994.  Tire  outcomes  reflect  different  aspects  or 
goals  of  training: 
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•  Reactions  address  training  content  and  design  (and  we  would 
argue,  delivery). 

•  Affective  outcomes  (i.e.,  attitudes)  and  learning  address  how  train¬ 
ing  results  in  changes  in  learners. 

•  Transfer  performance  and  results  address  how  training  brings  about 
payoffs  to  the  organization. 

Explanatory  factors  influence  or  point  to  reasons  for  training  out¬ 
comes  (e.g.,  students  with  low  levels  of  experience  do  not  benefit  from 
training  as  much  as  students  with  higher  levels  of  experience;  students 
with  a  “mastery”  learning  orientation  perform  better  than  students 
with  a  “performance”  orientation).'  Including  measures  of  such  factors 
can  improve  our  ability  to  predict  or  explain  adaptive  performance  and 
can  enable  designers  and  instructors  to  modify  aspects  of  the  course, 
such  as  prerequisites,  curriculum,  and  delivery,  to  improve  training 
outcomes. 

Our  evaluation  of  AWALP  included  measures  of  reactions,  affec¬ 
tive  outcomes,  learning,  and  application  of  course  principles  on  the 
job  as  a  proxy  for  transfer  performance.  The  substantive  focus  of  these 
measures  was  based  on  the  research  literature,  which  we  discuss  next. 

Research  Literature  Relevant  to  Measuring  AWALP  Effectiveness 

AWALP  addresses  adaptability  at  the  individual  level,  at  the  team  level, 
and  at  the  intersection  of  the  two — at  the  level  of  leadership.  We  briefly 
explore  the  research  literature  about  adaptability  at  each  of  these  levels 
and  explain  how  the  research  findings  informed  our  measures. 

Individual  Adaptability 
Background 

Although  numerous  authors  have  discussed  adaptability,  Pulakos  and 
her  colleagues  (Pulakos,  Arad,  et  ah,  2000;  Pulakos,  Schmitt,  et  ah. 


*  Mastery  and  performance  orientations  refer  to  motivational  dispositions  relevant  to  learn- 
ing  or  learning  goal  orientations  (Dweck,  1986).  Individuals  with  a  mastery  orientation  are 
interested  in  learning  for  the  sake  of  learning;  they  seek  learning  and  improvement.  Indi¬ 
viduals  with  a  performance  orientation  are  interested  in  learning  primarily  to  perform  better 
on  tasks;  they  are  interested  primarily  in  demonstrating  competence. 
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2002)  are  credited  with  synthesizing  ongoing  research  streams  under 
the  rubric  of  adaptability,  specihcally  in  the  service  of  defining  adaptive 
performance  (see  Baard,  Rench,  and  Kozlowski,  2014;  Ployhart  and 
Bliese,  2006).  Pulakos,  Arad,  et  ah,  2000,  built  on  the  well-established 
job  performance  model  put  forward  by  Campbell  and  his  colleagues 
that  emphasizes  a  focus  on  observable  behavior  and  that  breaks  perfor¬ 
mance  into  a  multidimensional  construct  (Campbell,  1990;  Campbell 
et  ah,  1993).  Although  they  attempted  to  model  the  full  breadth  of 
relevant  job  performance,  Campbell  and  his  colleagues  did  not  nec¬ 
essarily  include  a  component  in  their  model  that  covered  changes  in 
behavior  to  enable  adaptation — adaptive  performance  as  synthesized 
by  Pulakos,  Arad,  et  ah,  2000. 

In  the  Pulakos  et  al.  taxonomy,  adaptive  performance  is  conceived 
of  as  domain  general  rather  than  as  domain  specific — that  is,  perfor¬ 
mance  as  relevant  to  a  broad  range  of  tasks  (see  Baard,  Rench,  and 
Kozlowski,  2014).  Pulakos  and  her  colleagues  define  adaptive  perfor¬ 
mance  as  comprising  eight  factors,  as  described  in  Chapter  One.  In 
developing  their  taxonomy,  they  drew  from  the  organizational  litera¬ 
ture  and  from  an  extensive  database  of  critical  performance  incidents 
that  described  the  situation,  outcome,  and  conditions  surrounding 
instances  of  both  high  and  low  performance  in  a  wide  range  of  occu¬ 
pations  (Pulakos,  Arad,  et  ah,  2000).  The  design  of  AWALP — and  the 
measures  we  used  in  the  evaluation — are  based  on  these  dimensions  of 
adaptive  performance.  In  addition  to  identifying  dimensions  of  adap¬ 
tive  performance,  Pulakos,  Schmitt,  et  ah,  2002,  investigated  the  asso¬ 
ciation  of  potential  attitudinal  and  experiential  predictors  (self-efficacy, 
interest,  and  experience  measures  aligned  with  the  eight  dimensions) 
with  adaptive  job  performance  in  a  study  of  Army  personnel.  Their 
results  on  finding  separate  dimensions  in  supervisors’  ratings  of  adap¬ 
tive  performance  of  study  participants  were  equivocal;  for  supervisor 
ratings,  a  single  factor  was  found  to  fit  the  data  best.  However,  support 
was  found  for  distinctions  among  the  eight  factors  for  three  separate 
types  of  predictor  questions,  i.e.,  participants’  ratings  of  their  inter¬ 
est  in  adaptability,  self-efficacy  to  adapt,  and  experience  in  adapting. 
Moreover,  experience  in  adapting  added  incrementally  to  predicting 
supervisors’  ratings  of  adaptive  performance,  over  and  above  other  rel- 
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evant  and  well-established  traditional  predictors  of  job  performance 
(i.e.,  general  cognitive  ability  and  personality  traits,  including  achieve¬ 
ment  orientation  and  openness  to  experience), ^  although  experience 
in  only  one  adaptability  dimension  (i.e.,  learning  tasks,  technologies, 
and  procedures)  was  positively  and  significantly  associated  with  perfor¬ 
mance  ratings.^ 

More  generally,  there  is  a  large  body  of  research  documenting  the 
association  of  individual  characteristics  with  job  performance.  Studies 
show  consistent  and  positive  relationships  between  job  performance 
and  general  cognitive  ability  (e.g..  Hunter  and  Hunter,  1984;  LePine, 
Colquitt,  and  Erez,  2000;  Ree  and  Earles,  1991;  1992;  Ree,  Earles,  and 
Teachout,  1994;  Schmidt  and  Hunter,  2004);  learning  goal  orienta¬ 
tion  (e.g.,  Dweck,  1986;  Eisher  and  Eord,  1998;  Klein,  Noe  and  Wang, 
2006;  Mesmer-Magnus  and  Viswesvaran,  2007;  Phillips  and  Gully, 
1997);  and  personality  traits,  such  as  conscientiousness,  extraversion, 
and  openness  to  experience  (e.g.,  Barrick  and  Mount,  1991).  Research 
also  documents  the  association  of  individual  characteristics  with  leader 
effectiveness.  In  a  four-year  longitudinal  study  of  U.S.  Military  Acad¬ 
emy  cadets,  Bartone,  Snook,  and  Tremble,  2002,  found  that  gen¬ 
eral  intellectual  abilities  and  two  of  the  Big  five  personality  variables 
(agreeableness  and  conscientiousness)  predicted  leader  performance."^ 
Using  meta-analysis.  Judge  and  his  colleagues  have  also  found  that 
Big  five  personality  characteristics  predict  leader  performance.  Judge 
and  Bono,  2000,  showed  that  conscientiousness  and  extraversion  pre¬ 
dicted  transformational  leadership.  Judge  et  ak,  2002,  found  that,  of 
the  Big  five  traits,  extraversion  was  the  most  consistently  associated 
with  leader  emergence  and  leader  effectiveness,  followed  by  conscien¬ 
tiousness  and  openness  to  experience. 


^  Note,  however,  that  openness  was  negatively  related  to  performance. 

^  Experience  in  interpersonal  adaptability  was  also  significantly  associated  with  perfor¬ 
mance,  but  the  direction  of  the  correlation  was  negative. 

^  The  Big  Five  refers  to  five  personality  factors  considered  by  many  psychologists  to  com¬ 
prise  core  dimensions  of  personality.  The  five  dimensions  are  openness  to  experience,  consci¬ 
entiousness,  extraversion,  agreeableness,  and  emotional  stability. 
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Implications  for  Study  Design 

Our  study  did  not  include  quantitative  measures  of  adaptive  job  per¬ 
formance;  therefore,  our  evaluation  of  AWALP  at  the  individual  level 
focused  on  changes  in  attitudes  (self-efficacy,  interest,  and  experience) 
about  dimensions  of  adaptability  as  important  training  outcomes  in 
and  of  themselves  and  as  potential  proximal  indicators  of  adaptive  per¬ 
formance.  The  majority  of  measures  in  this  evaluation  focus  on  six 
of  the  original  eight  dimensions  of  adaptive  performance  (discussed 
in  Chapter  One)  emphasized  most  in  the  course:  thinking  creatively; 
dealing  with  changing  or  ambiguous  situations;  learning  tasks,  tech¬ 
nologies,  and  procedures;  interpersonal  adaptability;  cultural  adapt¬ 
ability;  and  decisionmaking  under  stress.  We  assessed  physical  adapt¬ 
ability  to  a  lesser  extent  and  assessed  handling  crisis  situations  to  a  very 
limited  extent. 

Because  of  their  potential  relevance  to  adaptive  performance,  we 
also  included  several  measures  of  individual  characteristics.  We  mea¬ 
sured  four  of  the  Big  Five  personality  traits,  including  extraversion, 
conscientiousness,  openness  to  experience,  and  agreeableness,  using 
the  NEO  Five-Factor  Inventory-3  personality  inventory  (Costa  and 
McCrae,  2010);  mastery  and  performance  goal  orientations  (Button, 
Mathieu,  and  Zajac,  1996);  general  cognitive  ability  using  the  Won- 
derlic  Contemporary  Cognitive  Ability  Test  (Wonderlic,  2012);  moti¬ 
vation  for  training  (i.e.,  perceived  benefits  of  participating  in  AWAFP); 
and  a  variety  of  demographic  characteristics. 

Team  Adaptability 
Background 

Fittle  work  in  organizations  today  is  accomplished  solely  at  the  indi¬ 
vidual  level;  very  often,  individuals  are  expected  to  come  together  in 
teams  to  accomplish  a  task.  While  there  is  an  abundance  of  studies  of 
team  effectiveness,  there  is  much  less  (albeit  increasing)  research  on 
the  nature  and  determinants  of  team  adaptability,  and  various  con¬ 
ceptualizations  of  team  adaptability  have  been  proposed.  For  example, 
Pulakos,  Dorsey,  and  White,  2006,  defined  team  adaptability  using  six 
of  the  eight  dimensions  of  individual  adaptability  (excluding  physical 
adaptability  and  cultural  adaptability)  and  postulated  individual  and 
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team  factors  that  contribute  to  team  adaptability.  Burke,  Stagl,  et  al., 
2006,  defined  team  adaptation  as  “a  change  in  team  performance, 
in  response  to  a  salient  cue  or  cue  stream,  that  leads  to  a  functional 
outcome  for  the  entire  team”  (p.  1190).  Chen,  Thomas,  and  Wallace, 
2005,  suggested  that  individual  and  team  adaptation  involve  the  trans¬ 
fer  of  newly  acquired  skills  and  knowledge  from  one  context  (training) 
to  another  more  complex  one  (a  more-complex  training  simulation). 
Kozlowski  et  ah,  1999,  suggested  that  adaptation  is  a  natural  by¬ 
product  of  successful  team  functioning  in  complex  environments,  in 
that  over  time  and  successive  cycles  of  performance,  individuals  come 
to  know  and  understand  their  places  on  a  team,  and  the  team  as  a  whole 
develops  an  understanding  of  which  team  members  (and  accompany¬ 
ing  knowledge,  skills,  abilities,  and  other  characteristics)  and  processes 
(such  as  coordination)  will  enable  the  team  to  react  to  the  environment 
productively.  The  team  develops  a  repertoire  of  flexible  responses  that  it 
can  apply  to  relevant  situations.  These  models  explore  in  varying  depth 
the  process  mechanisms  that  facilitate  or  impede  adaptation  and  the 
individual  differences  that  may  affect  those  processes. 

While  the  body  of  empirical  findings  regarding  team  adaptabil¬ 
ity  is  relatively  small,  it  is  well  established  that  team  processes,  such 
as  communication  and  coordination,  contribute  to  team  performance 
and  effectiveness  (e.g.,  Salas  et  ah,  2008),  including  performance  on 
tasks  that  are  novel,  complex,  stressful,  or  otherwise  require  change 
(i.e.,  require  adaptive  performance)  (e.g.,  Entin  and  Serfaty,  1999; 
Orasanu,  1990;  Smith-Jentsch,  Johnston,  and  Payne,  1998).  Moreover, 
team  training  affects  team  processes  (Salas  et  ah,  2008).  For  example, 
Entin  and  Serfaty,  1999,  showed  that  teams  can  be  trained  to  engage 
in  communication  and  coordination  behavior  that  contribute  to  shared 
mental  models  and  adaptive  performance. 

The  importance  of  teams  was  well  understood  in  the  design  of 
AWAEP.  Most  of  the  practical  exercises  are  conducted  in  teams  (with 
teams  being  recomposed  for  each  exercise).  The  curriculum  addresses 
such  concepts  as  the  distinction  between  taskwork  and  teamwork  (e.g., 
Morgan  et  ah,  1986),  and  emphasizes  processes,  such  as  communica¬ 
tion,  collaboration,  and  coordination,  to  foster  shared  mental  models 
and  situational  awareness.  In  addition  to  discussing  how  individuals 
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are  adaptable  in  the  context  of  team  activities,  a  central  focus  of  AARs 
is  how  adaptability  manifests  at  the  team  level. 

Implications  for  Study  Design 

Our  analysis  included  measures  of  the  dimensions  of  adaptability  at 
the  team  level.  We  asked  students  and  guides  to  assess  both  the  require¬ 
ments  for  the  dimensions  of  adaptive  performance  in  team  practical 
exercises  and  team  effectiveness  in  meeting  the  requirements.  In  addi¬ 
tion,  some  of  our  measures  assessed  individual  student  knowledge  of 
processes  associated  with  team  effectiveness. 

Leader  Adaptability 
Background 

At  the  intersection  of  the  individual  and  the  team  is  the  leader.  This, 
too,  has  been  the  focus  of  some  ongoing  work.  However,  Kozlowski 
et  ak,  2009,  noted  that,  while  team  leaders  are  the  key  to  developing 
team  capabilities,  this  aspect  of  their  role  is  largely  ignored  in  current 
leadership  theory. 

Noting  that  individual  adaptability  is  not  sufficient  with  regard 
to  Army  officers  in  particular.  White  et  ak,  2005,  suggested  adding  a 
ninth  dimension  to  the  initial  eight  Pulakos  and  colleagues  had  identi¬ 
fied:  leading  an  adaptable  team.  Tucker  and  Gunther,  2009,  collected 
critical  incidents  from  officers  and  NCOs  who  had  recently  deployed 
(whereas  those  in  Pulakos  et  ak’s  original  sample  were  collected  before 
these  conflicts  began).  Tucker  and  Gunther  found  that  officers,  in 
particular,  described  incidents  relating  to  dealing  with  ambiguous 
or  uncertain  environments,  although  relatively  few  critical  incidents 
related  to  the  new  dimension  of  leading  an  adaptable  team.  However, 
when  asked  for  suggestions  about  how  best  to  enhance  the  capacity  for 
adaptability,  the  number  of  incidents  pertaining  to  leading  adaptable 
teams  greatly  increased.  Other  work  (Zaccaro  et  ak,  2009)  suggests 
that  team-level  feedback  on  both  process  and  outcome  performance 
issues  enables  better  performance  (i.e.,  adaptive  behavior),  and  this 
work  notes  that  feedback  emphasizing  the  quality  of  the  team  inter¬ 
action  during  a  mission  is  likely  to  make  for  a  more- effective  AAR. 
Key  principles  in  AWALP  include  the  ideas  that  leader  training  should 
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focus  on  enhancing  team  development  and  that  effective  team  perfor¬ 
mance  can  enhance  leader  decisionmaking. 

Implications  for  Study  Design 

We  developed  questions  to  assess  the  intersection  of  the  leader  and  the 
team,  focusing  on  AWALP  participants’  self-efhcacy  for,  interest  in, 
and  perceived  need  for  behaviors  that  will  enable  them  to  develop  and 
lead  adaptable  teams.  We  refer  to  this  construct  as  “leader  practice.” 


Other  Methodological  Considerations 

Challenges  to  Measuring  Adaptability 

Adaptive  performance  is  difficult  to  measure  for  several  reasons.  First, 
the  dimensions  (e.g.,  handling  stress,  thinking  creatively)  are  largely 
intangible  or  abstract,  and  intangible  concepts  are  typically  measured 
using  self-report  methods,  which  can  be  subject  to  biases  and  threats 
to  validity. 

Some  of  these  potential  biases  and  threats  to  validity  are  illus¬ 
trated  in  the  results  obtained  from  the  method  used  to  assess  AWALP 
in  the  past.  In  brief,  at  the  start  of  training,  students  were  asked  to 
respond  to  ten  questions  about  their  level  of  adaptability  readiness.  For 
example,  students  were  asked  if  they  were  prepared  or  unprepared  with 
respect  to  “Understanding  of  what  it  means  to  be  adaptable.”  At  the 
conclusion  of  training,  students  were  provided  with  their  pretraining 
responses  and  asked  to  report  how  they  would  have  answered,  know¬ 
ing  what  they  know  now.  They  were  then  asked  to  report  their  current 
level  of  preparation  having  completed  AWALP.  Figure  2.1  provides  an 
example  of  these  responses. 

As  shown  in  Figure  2.1,  nearly  all  the  students  felt  that  they 
understood  what  adaptability  meant  prior  to  training.  After  training, 
the  majority  realized  that  they  were  not  prepared  initially,  and  all  stu¬ 
dents  felt  prepared  moving  forward.  The  pattern  of  results  was  similar 
across  the  ten  items  and  suggests  that  students  believe  AWALP  was 
highly  effective  in  training  soldiers  to  be  adaptable. 
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Figure  2.1 

Example  of  Results  from  Prior  AWALP  Evaluation 


Understanding  of  what  it  means  to  be  adaptable 


I  I  Prepared 
I  I  Unprepared 


RAND  RR504-2. 1 


However,  there  are  a  variety  of  alternative  interpretations  for  these 
results: 

•  Questions  are  likely  subject  to  social  desirability  bias  in  that,  prior 
to  training,  respondents  may  not  want  to  admit  that  they  do  not 
know  what  it  means  to  he  adaptable  or  may  want  to  give  the 
impression  that  they  are  adaptable. 

•  The  general  nature  of  questions,  such  as  knowing  what  it  means 
to  be  adaptable,  may  not  be  very  instructive  because  respondents 
may  not  know  what  they  do  not  know.  In  addition,  similar  pat¬ 
terns  of  results  across  items  suggest  that  the  responses  do  not  dis¬ 
criminate  among  different  constructs  addressed  in  the  course. 

•  The  high  ratings  before  AWALP  began  result  in  ceiling  effects, 
making  it  difhcult  to  demonstrate  improvement  or  to  document 
associations  of  responses  with  other  variables,  such  as  individual 
differences  or  other  outcome  measures. 

•  Providing  students  with  their  pretraining  responses  and  asking 
them  to  reflect  back  after  training  presents  demand  characteris¬ 
tics  (i.e.,  when  respondents  become  aware  of  what  the  questions 
are  investigating  and  change  their  behavior).  Thus,  respondents 
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essentially  know  what  the  “expected”  answers  are  and  may  have 

responded  accordingly. 

A  second  reason  adaptability  is  difficult  to  measure  is  that  a  com¬ 
prehensive  assessment  can  pose  a  signihcant  response  burden,  particu¬ 
larly  because  adaptability  can  occur  at  different  levels  (i.e.,  individual, 
team,  and  the  leader-team  intersection).  Assessing  change  due  to  the 
training  requires  administering  pretraining  measures,  further  increas¬ 
ing  the  response  burden.  Furthermore,  because  adaptability  is  likely 
correlated  with  a  variety  of  individual  characteristics,  it  is  important  to 
measure  the  characteristics  to  assess  whether  the  impact  of  an  interven¬ 
tion,  such  as  AWALP,  occurs  in  addition  to  the  effects  of  the  character¬ 
istics — that  is,  whether  the  intervention  itself  has  an  identifiable  effect 
beyond  characteristics  that  would  predict  adaptable  behavior.  How¬ 
ever,  this  again  adds  to  the  measurement  burden. 

Finally,  although  there  are  many  established  and  available  mea¬ 
sures  of  individual  characteristics  that  are  likely  to  be  associated  with 
adaptive  performance,  few  measures  of  adaptive  performance  are  avail¬ 
able  in  the  public  domain.  Developing  reliable  and  valid  measures 
requires  a  large  number  of  respondents. 

Addressing  Challenges  to  Measuring  Adaptability 

We  used  a  fourfold  strategy  to  address  these  challenges.  First,  we  used 
a  combination  of  multiple  measures  of  adaptive  performance,  includ¬ 
ing  piloting  new  ones.  The  goals  of  doing  so  were  to  assess  a  range  of 
adaptability  training  outcomes  and  to  offset  the  weaknesses  of  any  one 
method  or  measure,  which  might  be  subject  to  a  particular  bias,  with 
other  approaches. 

Second,  rather  than  asking  general  questions  about  students’  self¬ 
perceptions  of  adaptability  knowledge  or  skills,  we  asked  about  specific 
behaviors  that  are  indicative  of  the  adaptive  performance  dimensions. 
This  approach  was  used  to  provide  precise  terms  that  are  consistent 
with  the  Pulakos  et  ah,  2000,  taxonomy  and  to  attempt  to  reduce 
demand  characteristics  and  social  desirability  biases.  We  also  used  an 
objective  measure  of  knowledge  in  the  form  of  a  multiple-choice  test  on 
adaptability  and  team  concepts. 
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Third,  for  most  outcome  measures,  we  used  a  pretest-posttest 
design,  rather  than  a  posttest-only  design,  and  included  measures  of 
individual  differences  that  we  anticipate  are  associated  with  adaptive 
performance.  This  approach  enabled  us  to  draw  inferences  about  the 
impact  of  the  intervention  (AWALP  training). 

Fourth,  to  balance  our  relatively  comprehensive  approach  with 
response  burden,  we  focused  our  evaluation  on  the  factors  we  con¬ 
sider  most  critical  to  adaptive  performance  and  to  the  AWALP  course. 
For  example,  we  assessed  adaptability  primarily  for  individuals  and  to 
a  lesser  extent  for  teams,  given  that  the  goal  of  the  course  is  to  train 
adaptability  at  the  individual  and  leader  level  rather  than  to  train  intact 
teams  through  multiple  experiences.  Furthermore,  we  included  some, 
but  not  all,  of  the  dimensions  of  adaptive  performance  in  our  instru¬ 
ments,  in  large  part  reflecting  the  emphasis  of  each  dimension  in  the 
course  as  described  in  Chapter  One. 


Study  Design 

Participants 

The  evaluation  of  AWALP  focused  on  students  enrolled  in  three 
AWALP  classes  conducted  in  January,  April,  and  August  2013  (Class 
10,  11,  and  12,  respectively).  There  were  104  students  across  the  three 
classes.  With  the  exception  of  interviews,  described  in  Chapter  Five, 
participation  rates  in  the  evaluation  study  were  close  to  100  percent; 
three  students  did  not  complete  the  surveys  or  tests  because  they 
arrived  late  to  the  course  or  left  early.  Some  students  did  not  answer 
all  the  questions;  therefore,  we  have  missing  values  for  some  analyses. 

On  average,  student  participants  were  31.5  years  old  {SD  =  6.28) 
and  had  10.83  years  of  service  {SD  =  5.68).  Table  2.1  shows  other 
demographic  characteristics  of  the  students. 

Nine  of  the  students  in  one  of  the  AWALP  classes  were  opera¬ 
tional  advisors,  who  were  included  in  the  class  as  part  of  their  train¬ 
ing  to  work  with  Army  and  joint  force  units  to  predict  and  respond  to 
asymmetric  threats  and  methods.  Operational  advisors’  responses  to 
the  measures  used  in  the  evaluation  did  not  differ  from  other  students. 
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Table  2.1 

Student  Demographic  Characteristics 


Category 

Demographic 

Percent 

Rank 

Specialist  or  sergeant  (E4-5) 

15 

Staff  sergeant  (E6) 

31 

Sergeant  first  class  (E7) 

25 

Master  sergeant,  first  sergeant,  sergeant  major,  or 
command  sergeant  major  (E8-9) 

18 

Officer  (2LT-LTC) 

12 

Career  field 

Infantry 

42 

Armor 

13 

Field  Artillery 

12 

Military  Intelligence 

9 

Military  Police  Corps 

7 

Other 

17 

Highest  degree 

High  school  or  GED 

18 

Some  college 

46 

Two-year  college  degree 

10 

Bachelor's  or  graduate  degree 

26 

NOTE:  Percentages  may  not  sum  to  100  due  to  rounding. 


SO  they  were  included  in  the  analyses.  However,  they  were  not  included 
in  the  assessment  of  graduates’  application  of  AWALP  principles  on  the 
joh  reported  in  Chapter  Five. 

In  addition  to  student  participants,  seven  guides  participated 
in  one  component  of  the  research  in  which  they  evaluated  team  per¬ 
formance  effectiveness,  described  in  more  detail  helow  and  in  Chap¬ 
ter  Four.  Supervisors  participated  in  follow-up  interviews,  which  are 
described  in  Chapter  Five. 


Table  2.2 

Measures,  Methods,  Constructs,  and  Schedule  of  Administration 


Class  Administered 

Measure 

Methods 

Timing 

Constructs 

10 

11 

12 

Reactions 

Survey 

Posttraining 

Satisfaction  with  training 

• 

• 

• 

Affective 

training 

outcomes 

(attitudes) 

Survey 

Pre-  and  posttraining 

Self-efficacy  for  behaviors  associated  with 
adaptive  performance  and  leading  adaptable 
teams,  interest  in  engaging  in  these  behaviors, 
and  frequency  of  engaging  in  these  behaviors 

• 

• 

• 

Learning 

(cognitive) 

Paper-and-pencil  test 

Pre-  and  posttraining 

Declarative  knowledge  of  adaptability  and 
team  concepts 

• 

• 

• 

Learning 

(behavioral) 

Survey  of  peer  ratings 
of  individuals 

After  phase  1  and 
phase  2 

Requirements  for  and  effectiveness  of 
adaptive  performance 

• 

Survey — students' 
ratings  of  teams 

After  exercises  on  days 
3,  5,  and  7 

• 

• 

Survey — guides' 
ratings  of  teams 

• 

• 

Application  of 
AWALP  concepts 
on  the  job 

Interviews 

Three  and  six  months 
posttraining 

Graduate  and  supervisor  reports 

• 

• 

Individual 

characteristics 

Paper-and-pencil  test; 
survey 

Pretraining 

General  cognitive  ability 

Personality  traits 

Learning  goal  orientation 

Motivation  for  training  (perceived  benefits  of 
AWALP) 

Demographic  characteristics 

• 

• 

• 

General  Approach  and  Study  Design  23 
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Methods,  Measures,  and  Procedures 

Table  2.2  shows  the  measures,  methods,  and  constructs  measured  in 
the  evaluation,  along  with  the  schedule  of  administration  of  the  mea¬ 
sures.  Pretraining  measures  were  collected  the  evening  before  AWALP 
began  (“day  0”),  and  posttraining  measures  were  collected  on  the  last 
day  of  AWALP  (“day  10”).  Data  collection  took  approximately  80  min¬ 
utes  on  day  0  and  45  minutes  on  day  10.  As  shown  in  Table  2.2,  other 
outcomes  (peer  ratings  and  ratings  of  teams)  were  collected  at  various 
points  throughout  the  course,  and  application  of  course  principles  on 
the  job  was  measured  three  or  six  months  following  AWALP.  Measures 
were  administered  using  paper-and-pencil  instruments.  For  most  mea¬ 
sures,  there  were  no  differences  by  class,  so  analyses  were  conducted 
across  all  three  classes. 

Some  of  the  individual  characteristics  measures  (general  cognitive 
ability,  personality  traits)  are  from  commercial  instruments.  We  used 
these  measures  in  all  three  classes.  In  contrast,  we  took  an  experimental 
approach  with  measures  we  developed  for  this  evaluation.  For  exam¬ 
ple,  we  modified  methods  (e.g.,  for  attitudes)  or  discontinued  measures 
(e.g.,  peer  ratings)  after  learning  from  the  experience  of  the  first  class. 
Furthermore,  we  did  not  use  a  measure  (e.g.,  team  ratings)  in  every 
class  to  balance  the  comprehensiveness  of  the  evaluation  with  response 
burden. 

The  commercial  instruments  used  in  the  evaluation  are  publicly 
available.  However,  to  maintain  the  integrity  of  future  evaluations 
of  AWALP,  neither  the  knowledge  test  nor  the  measures  of  attitudes 
toward  adaptability  that  we  developed  for  this  evaluation  are  pub¬ 
lished.  These  measures  may  be  obtained  by  contacting  AWG. 

In  Chapters  Three  through  Five,  we  describe  the  methods  and 
measures  in  more  detail  and  present  the  results  of  the  evaluation  for  the 
measures  shown  in  column  1  of  Table  2.2. 


CHAPTER  THREE 


Reactions  Toward  AWALP  and  Attitudes  Toward 
Adaptability 


This  chapter  reports  on  reactions  to  the  course,  measured  at  the  end  of 
training,  and  changes  in  attitudes  toward  adaptive  performance,  mea¬ 
sured  at  the  beginning  and  end  of  the  course. 


Reactions  to  AWALP 

Method 

Reactions  were  measured  in  the  posttraining  survey.  The  survey 
included  28  close-ended  questions  assessing  students’  views  of  course 
content,  delivery,  and  structure  and  six  open-ended  questions  asking 
students  to  elaborate  on  some  of  their  responses.  Examples  of  close- 
ended  questions  include  “AWALP  guides  effectively  facilitated  course 
exercises,”  and  “Attending  AWALP  was  a  good  use  of  my  time.”  Some 
of  the  questions  were  based  on  White  et  ah,  2005,  and  Straus,  Shanley, 
Yeung,  et  ah,  2011.  Most  of  the  close-ended  questions  used  six-point 
response  scales,  with  options  ranging  from  1  =  strongly  disagree  to 
6  =  strongly  agree.  A  small  number  of  questions  provided  response 
scales  with  three  options  (e.g.,  too  much,  about  right,  too  little;  see 
Ligure  3.2).  The  questions  are  shown  in  Appendix  B. 

Results 

Students  reported  high  levels  of  satisfaction  with  AWALP. 

figure  3.1  shows  the  average  scores  on  responses  to  items  (a  number 
of  which  were  a  composite  of  multiple  survey  questions)  about  course 
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Figure  3.1 

Average  Ratings:  Course  Delivery  and  Overall  Satisfaction 


Overall  satisfaction 

AWALP  contributed 
to  my  learning 

Training  facilities 
Course  materials 

Peer  feedback 

Guide  feedback — 
team  performance 
Guide  feedback — 
individual  performance 
Guide  knowledge  and 
facilitation 


Rating 


NOTES:  Some  aspects  of  satisfaction  were  measured  with  multiple-item  scales, 
including  guide  knowledge  and  facilitation  (three  items),  AWALP  contributed  to 
learning  (1 1  items),  and  overall  satisfaction  (four  items).  Coefficient  a  values  for  these 
scales  were  0.79,  0.93,  and  0.88,  respectively.  Coefficient  a  is  a  measure  of  the  internal 
consistency  reliability  of  a  set  of  items.  It  typically  ranges  from  zero  to  one,  with 
higher  values  indicating  greater  reliability.  Coefficient  a  values  of  0.80  and  higher  are 
generally  considered  satisfactory,  although  values  are  influenced  by  the  number  of 
items  on  a  scale;  thus,  a  scale  with  a  few  items  will  typically  have  a  lower  a  than  a 
scale  with  many  items. 
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delivery  and  overall  satisfaction  in  the  course.  Average  scores  ranged 
from  approximately  5  to  6.  The  highest  rating  was  given  to  the  guides 
(guide  knowledge  and  facilitation),  closely  followed  hy  several  other 
aspects  of  course  delivery  and  overall  satisfaction.  Feedback  from 
guides  about  team  performance  was  also  given  high  ratings,  but  rat¬ 
ings  for  guide  feedback  about  individual  performance  were  lower,  sug¬ 
gesting  that  the  course  focuses  more  on  the  former. 

“AWALP  contributed  to  my  learning”  (the  second  bar  from  the 
top  in  Figure  3.1)  is  a  combination  of  answers  to  items  dealing  with 
learning  different  dimensions  of  adaptive  performance  and  various 
leader  practices.  The  average  rating  was  5.50,  and  each  component 
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question  also  had  a  high  rating,  ranging  from  about  5.27  to  5.60.  To 
gain  additional  insight  about  how  the  course  contributed  to  learning, 
a  follow-up,  open-ended  question  asked  students  to  identify  the  two 
topics  for  which  they  changed  the  most  and  the  two  topics  for  which 
they  changed  the  least  as  a  result  of  the  course;  it  further  asked  them 
to  explain  the  reasons  for  their  choices.  Frequencies  of  “top  two”  and 
“bottom  two”  responses  are  shown  in  Figure  3.2. 

While  the  areas  of  most  and  least  improvement  were  distributed 
across  all  response  options,  AARs  stand  out.  Fifty  percent  of  the  stu¬ 
dents  identified  AARs  as  one  of  the  two  areas  in  which  they  learned 
the  most.  The  most  frequent  explanations  for  “why”  included  (in  order 
of  frequency)  that  the  AWALP  method  promotes  better  communica¬ 
tion,  provides  a  better  way  to  plan,  facilitates  the  use  of  open-ended 
questions,  allows  teams  to  go  deeper,  and  promotes  a  focus  on  learn¬ 
ing.  Creative  thinking,  identified  by  25  percent  of  students,  was  also  a 
frequently  mentioned  “top  two”  area.  The  most  frequently  mentioned 
comments  for  creative  thinking  were  that  AWALP  taught  graduates 
to  keep  an  open  mind  and  how  to  correctly  analyze  problems.  Among 

Figure  3.2 

Areas  of  Most  and  Least  Improvement 
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“bottom  two”  topics,  cultural  interaction  was  the  area  most  frequently 
mentioned  (by  over  30  percent  of  students).  By  far,  the  most  frequently 
mentioned  reason  for  this  choice  (not  only  in  the  cultural  area  but  in 
all  areas  of  least  learning)  was  that  students  felt  that  they  already  had 
the  skills  or  attribute  prior  to  attending  AWALP  (e.g.,  “Interacting  with 
different  cultures  was  already  a  strong  point”). 

All  students  reported  that  they  would  recommend  AWALP 
to  others;  the  most  common  reasons  were  the  course  content  and 
training  approach.  After  students  reported  whether  they  would  rec¬ 
ommend  AWALP  to  others,  which  was  one  of  the  questions  on  the 
“Overall  satisfaction”  scale,  they  were  then  asked  to  respond  to  an 
open-ended  question  about  why  or  why  not  (note  that  all  students 
gave  an  “agree”  response  to  this  question).  We  used  content  analysis 
to  group  responses  into  categories.  Results  are  shown  in  Figure  3.3. 
Among  those  who  answered  the  question,  the  most  common  reason 
for  recommending  the  course  was  because  of  the  adaptability  con¬ 
cepts  they  learned  (e.g.,  “learned  effective  AAR,”  and  “fostered  creative 
thinking”).  Other  common  responses  were  that  students  valued  the 
training  approach  (e.g.,  “introduces  an  entirely  new  and  more  benefi- 

Figure  3.3 

Reasons  Students  Would  Recommend  AWALP  to  Others 
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dal  teaching/learning  style”)  or  thought  that  AWALP  improved  their 
cognition  (e.g.,  “learned  better  techniques  to  think”  and  “makes  you 
open  your  mind”).  Eleven  percent  of  the  students  responded  that  what 
they  learned  in  the  course  will  help  them  develop  in  their  careers  (e.g., 
“it  helps  develop  leadership  traits  rarely,  if  ever,  trained  outside  [of 
AWALP]”).  Finally,  3  percent  of  the  students  said  they  would  recom¬ 
mend  the  course  based  on  the  quality  of  the  guides.* 

Students  were  highly  satisfied  with  the  course  structure. 
Figure  3.4  shows  frequencies  of  responses  to  questions  about  the  course 
structure.  Results  show  that  the  preponderance  of  students  thought  the 
course  was  “about  right”  with  respect  to  course  length,  classroom  time, 
difficulty  level  of  course  content,  and  difficulty  of  course  exercises.  The 
area  with  the  greatest  perceived  need  for  change  was  in  course  length, 
for  which  13  percent  of  students  thought  that  the  course  should  be 

Figure  3.4 

Ratings  of  Course  Structure 


I  I  Too  short 
I  I  About  right 
Too  long 


I  I  Too  little 
I  I  About  right 
Too  much 


I  I  Too  academic 
I  I  About  right 
Mostly  common 
sense 


NOTE:  Questions  based  on  White  et  al.,  2005. 
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I  I  Too  difficult 

I  I  About  right 

Too  easy 


^  The  “other”  category  primarily  comprised  comments  that,  rather  than  providing  specific 
reasons  for  recommending  the  course,  demonstrate  respondents’  enthusiasm  for  the  course 
by  recommending  widespread  Army  attendance  or  reaffirming  that  students  thought  “it  was 
a  great  course.” 
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longer.  As  the  results  below  suggest,  requests  for  a  longer  course  can 
be  interpreted  as  a  positive  sentiment  about  the  course;  this  interpre¬ 
tation  is  also  consistent  with  patterns  of  overall  satisfaction  ratings, 
which  were  very  favorable  among  all  students  (see  Figure  3.1),  but  were 
significantly  higher  for  students  who  thought  that  the  course  should 
be  longer  than  for  students  who  though  the  course  was  “about  right.”^ 
Recommendations  for  improvements  to  AWALP  were  directed 
largely  at  making  the  course  even  better,  as  opposed  to  identify¬ 
ing  and  resolving  weaknesses.  Figure  3.5  shows  results  of  analyz¬ 
ing  responses  to  an  open-ended  question  asking  students  for  suggested 
changes  to  the  course,  which  were  coded  into  categories  representing 
common  themes.  Twenty-eight  percent  of  students  reported  no  need 
for  change,  and  24  percent  did  not  answer  the  question.  In  the  “change 
training”  (17  percent)  and  “add  training”  (12  percent)  columns,  the 
majority  of  responses  were  general  (e.g.,  “maybe  more  shooting”  for 
change  training  and  “make  the  course  longer”  for  add  training),  but 
the  overall  theme  of  these  comments  was  to  enhance  an  already  effec¬ 
tive  course.  Nine  percent  of  the  students  recommended  that  students 
remove  rank  and  patches  during  the  course  to  put  all  participants  on 

Figure  3.5 
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a  level  playing  field;  as  one  student  put  it,  “to  let  everyone  know  that 
rank  has  nothing  to  do  with  one’s  creative  thinking.”  The  “other”  cate¬ 
gory  contains  responses  that  were  not  specifically  about  course  content 
or  delivery;  for  example,  one  student  suggested  being  more  selective 
about  who  attends. 

Other  common  requests  for  improvements  can  be  found  in  the 
“change  training”  category.  In  particular,  five  students  suggested 
making  the  capstone  exercise  more  challenging  (e.g.,  “[add]  more 
stress”;  “did  not  push  threshold  of  failure”;  “more  time,  more  freedom 
of  movement,  more  negative  reactions”;  “[make]  real  world,  with  real 
intelligence  gathered”).  Three  recommended  encouraging  more  indi¬ 
vidual  involvement  in  training  exercises  (e.g.,  “[give]  more  individu¬ 
als  more  opportunities  to  be  put  on  the  spot,  i.e.,  to  speak,  lead,  solve 
problems”).  Only  two  students  thought  the  course  was  too  long. 

Students  anticipated  that  leader  buy-in  and  command  cli¬ 
mate  would  pose  challenges  to  applying  adaptability  concepts  on 
the  job.  Students  were  asked  in  an  open-ended  question  about  chal¬ 
lenges  they  expected  in  applying  AWALP  concepts.  Responses  were 
coded  into  categories  based  on  common  themes.  Nearly  two-thirds 
of  the  students  identified  one  or  more  challenges.  About  17  percent  of 
the  students  said  they  anticipated  no  challenges,  and  an  equal  number 
skipped  the  question. 

Leader  buy-in  was  the  most  frequently  mentioned  challenge,  a 
response  identified  by  nearly  one-third  of  the  students  (see  Figure  3.6). 
Some  expressed  this  sentiment  quite  succinctly  (e.g.,  my  challenge  is 
“my  higher  chain  of  command).”  Others  provided  more  details.  For 
example,  one  student  said  “I  will  want  to  push  new  ideas  too  quickly 
into  my  training  plans.  My  biggest  challenge  will  be  to  pace  that  and 
sell  my  leadership  on  what  I  am  trying  to  accomplish.” 

Twenty  percent  of  students  responded,  somewhat  more  gener¬ 
ally,  that  they  anticipated  “command  climate”  to  be  a  challenge.  For 
example,  one  student  talked  about  the  need  for  “fostering  command 
climate.”  Another  said  “there  is  the  challenge  of  changing  the  Army 
culture  to  open  up  to  this  way  of  training.”  Collapsing  “leadership  buy- 
in”  and  “command  climate”  responses,  over  one-half  of  the  graduates 
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Figure  3.6 

Anticipated  Challenges  to  Implementing  AWALP  Principles 
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expressed  the  sentiment  that  it  is  a  challenge  to  get  the  Army  to  accept 
the  AWALP  way  of  training. 

Seven  percent  of  students  anticipated  challenges  designing  new 
training.  One  student  commented,  “My  first  few  training  ideas  will 
be  the  most  difficult  to  construct.  I  will  build  from  those  successes/ 
failures  in  order  to  create  better  ambiguous  situations  and  foster  learn¬ 
ing  environments.”  We  also  categorized  5  percent  of  the  responses  into 
an  “other”  category.  For  example,  one  graduate  commented  that  there 
would  be  time  and  resource  limitations  that  would  restrict  the  design 
of  AWALP  training. 

Despite  the  challenges  that  students  foresaw,  we  also  detected  a 
sense  of  determination  and  spirit  in  responses  expressed  by  almost  one- 
third  of  students.  Although  not  shown  in  Figure  3.6,  13  students  vol¬ 
unteered  that  they  would  personally  take  on  anticipated  challenges, 
working  to  persuade  others  of  the  value  of  implementing  AWALP  prin¬ 
ciples.  Eleven  students  went  further,  expressing  confidence  that  the 
challenges  could  be  overcome,  sometimes  by  persuading  others  and 
sometimes  by  their  own  determination,  e.g.,  “the  only  real  challenge  is 
how  creative  I  am  willing  to  get.” 
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Summary  of  Reactions  to  Training 

Students  had  extremely  favorable  reactions  to  AWALP,  both  overall  and 
with  respect  to  specific  aspects  of  the  course.  Furthermore,  students 
indicated  that  they  would  recommend  the  course  to  others  precisely 
because  of  its  unique  teaching  approach.  Even  among  the  minority  of 
students  who  suggested  course  improvements,  most  indicated  that  they 
wanted  to  make  a  good  course  even  better.  Despite  these  positive  reac¬ 
tions,  nearly  two-thirds  of  the  students  saw  challenges  in  applying  the 
principles  they  learned  in  AWALP  when  they  returned  to  their  units. 

It  is  worth  noting  that  reactions  to  AWALP  are  much  more  favor¬ 
able  than  reactions  to  Army  institutional  leader  training  reported  else¬ 
where.  Lor  example,  the  Center  for  Army  Leadership  Annual  Survey 
of  Army  Leadership  (CASAL)  (Riley  et  ah,  2013)  reported  that  in 
response  to  a  global  question  about  institutional  training  in  2012, 
58  percent  of  active-duty  leaders  rated  the  training  as  very  effective 
or  effective  in  preparing  them  for  leadership,  and  20  percent  rated  it 
as  ineffective.  In  response  to  questions  about  specific  Army  courses 
and  schools  from  2008  to  2012,  46  percent  of  respondents  reported 
that  courses  improved  their  leadership  capabilities;  53  percent  found 
the  content  relevant  to  leadership  responsibilities  in  their  next  job;  and 
65  percent  found  the  content  was  up  to  date.  Although  these  questions 
are  not  identical  to  those  asked  in  the  AWALP  evaluation,  it  is  clear 
that  the  degree  of  satisfaction  with  AWALP  is  substantially  higher. 


Attitudes  Toward  Adaptive  Performance 

Method 

We  assessed  affective  training  outcomes  by  measuring  attitudes  toward 
adaptive  performance  and  changes  in  the  attitudes  using  self-report 
measures  administered  pre-  and  posttraining.  We  used  or  adapted 
published  items  from  Ployhart  and  Bliese,  2006;  Pulakos  et  ah,  2000; 
Pulakos  et  ah,  2002;  and  White  et  ah,  2005,  and  supplemented  them 
with  original  items.  Items  measured  six  of  the  eight  dimensions  of 
adaptive  performance  (creative  thinking;  dealing  with  ambiguous  sit¬ 
uations;  learning  tasks,  technologies,  and  procedures;  interpersonal 
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adaptability;  cultural  adaptability;  and  decisionmaking  under  stress). 
The  survey  included  two  or  three  items  about  each  of  these  dimensions. 
Examples  include  “Come  up  with  alternative  courses  of  action  that 
others  may  not  have  considered”  (for  creative  thinking)  and  “Deal  with 
a  situation  where  things  are  not  ‘black  and  white’”  (for  dealing  with 
ambiguous  situations).  In  addition,  based  on  topics  addressed  in  the 
AWALP  curriculum,  we  developed  seven  items  about  putting  adapt¬ 
ability  concepts  into  leader  practice,  such  as  “Coach  subordinates,” 
“Delegate  responsibility  to  others  on  tasks  on  which  they  might  fail,” 
and  “Seek  input  from  subordinates  about  a  complex  decision.” 

Note  that  the  items  do  not  ask  directly  about  adaptability  in 
AWALP  or  a  specific  adaptive  performance  dimension;  rather,  they  ask 
about  a  variety  of  more-general  situations  that  relate  to  those  dimen¬ 
sions.  The  goals  of  this  approach  are  to  reduce  social  desirability  biases 
and  potential  ceiling  effects,  demand  characteristics  (responding  with 
what  respondents  see  as  the  “expected”  answer),  and  other  issues  related 
to  students  not  necessarily  knowing  what  adaptability  means. 

Following  Pulakos  et  ak,  2002,  we  measured  three  aspects 
or  attributes  of  attitudes:  (1)  frequency  with  which  students  engage 
in  each  behavior  or  situation  in  their  jobs  (pretraining)  or  feel  that 
they  should  engage  in  the  behavior  or  situation  in  their  jobs  (post¬ 
training),  (2)  interest  in  engaging  in  the  behavior  or  situation,  and 
(3)  self-efficacy  for  the  behavior  or  situation.^  Unlike  Pulakos  et  ak, 
2002,  which  examined  whether  experience,  interest,  and  self-efficacy 
predicted  performance  ratings  by  supervisors,  we  administered  the 
questions  before  and  after  training  to  assess  changes  in  attitudes  follow¬ 
ing  AWALP.  Accordingly,  we  revised  the  posttraining  question  about 
experience  or  frequency  to  reflect  the  need  for  adaptability  behaviors. 

Response  options  for  questions  about  frequency  included  “never,” 
“a  few  times/year,”  “monthly,”  “weekly,”  and  “daily.”  Response  options 
for  questions  about  interest  and  self-efficacy  used  six-point  scales  rang- 


^  In  the  first  class  of  students,  we  also  asked  about  the  importance  of  engaging  in  each 
behavior  or  activity.  Results  showed  that  most  students  thought  adaptability  was  important 
to  their  jobs  coming  into  the  course;  thus,  there  were  ceiling  effects  for  responses  prior  to 
training,  leaving  little  room  for  improvement.  Consequently,  we  eliminated  this  attribute  for 
subsequent  classes  to  reduce  response  burden. 
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ing  from  1  =  strongly  disagree  to  6  =  strongly  agree.  Table  3.1  shows 
the  format  for  the  questions.^ 

For  measures  of  frequency,  we  compared  the  distribution  of  the 
differences  between  responses  pre-  and  posttraining  using  an  exam¬ 
ple  item  of  each  adaptive  performance  dimension.^  For  interest  and 
self-efficacy,  we  present  the  mean  ratings  of  the  items  measuring  each 
dimension  and  leader  practice.  Statistical  tests  were  directional,  reflect¬ 
ing  the  expectation  that  attitudes  would  become  more  positive  from 
pre-  to  posttraining.  We  also  adjusted  the  alpha  level  for  multiple  tests 
using  a  Bonferroni  correction. 

Results 

There  was  improvement  in  almost  all  aspects  of  attitudes  toward 
adaptive  performance.  As  Table  3.2  shows,  after  the  course,  students 


Table  3.1 

Format  for  Questions  About  Attitudes  Toward  Adaptive  Performance  Pre- 
and  Posttraining 


Attribute 

Instructions 

Frequency 

How  often  is  this  activity  a  part  of  your  job?  (pretraining) 

How  often  do  you  think  you  should  be  doing  this  activity  on  your 
job?  (posttraining) 

Interest 

1  would  like  doing  this  task  or  working  in  situations  that  require  this 
activity  (pre-  and  posttraining) 

Self-efficacy 

1  am  capable  of  carrying  out  each  activity,  today,  without  any 
additional  training  (pre-  and  posttraining) 

^  Coefficient  a  for  the  self-efficacy  and  interest  scales  was  generally  high,  ranging  from 
0.65  to  0.87,  indicating  that  the  scales  were  internally  consistent.  The  Likert-type  response 
scales  used  for  these  questions  can  approximate  a  continuous  variable;  however,  coefficient  (X 
is  not  applicable  for  the  frequency  scales,  which  used  such  response  options  as  “never”  and 
“monthly,”  and  reactions  to  course  structure  in  the  current  evaluation,  which  used  such 
response  options  as  “too  much,”  “too  little,”  and  “about  right”). 

^  Only  one  item  was  used  because  responses  to  frequencies  cannot  be  added  together  to 
create  a  single  measure.  However,  we  analyzed  all  the  items  to  verify  that  results  were  simi¬ 
lar  for  the  different  items  measuring  each  construct.  This  was  the  case  for  the  dimensions  of 
adaptability,  but  there  were  differences  in  the  distributions  of  answers  for  the  items  measur¬ 
ing  leader  practice.  Therefore,  we  report  the  leader  practice  items  separately. 
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Table  3.2 

Summary  of  Changes  in  Attitudes 


Dimension 

Seif-efficacy 

Interest  in 
performing 

Need  in 
current  job 

Creative  thinking 

X 

X 

X 

Ambiguous  situations 

X 

X 

X 

Learning 

X 

X 

X 

Interpersonal 

X 

X 

X 

Cultural 

X 

X 

X 

Decisionmaking  under  stress 

X 

X 

X 

Leader  practice 

X 

X 

Varied 

thought  they  were  better  at  being  adaptable  in  all  dimensions  and 
in  leader  practice.  They  also  reported  greater  interest  in  adaptability 
than  they  had  before  the  course.  In  addition,  at  the  end  of  the  course, 
students  saw  their  jobs  as  requiring  more  adaptive  performance  with 
respect  to  all  the  dimensions  of  adaptability  and  most  leader  practice 
behaviors.  Thus,  after  the  course,  students  not  only  sensed  a  change  in 
their  own  capabilities  and  interests  regarding  adaptive  performance, 
but  they  saw  their  work  contexts  in  a  different  light. 

The  following  sections  present  our  results  for  each  of  the  three 
attributes  of  students’  attitudes. 

Self-Efficacy  for  Adaptive  Performance 

Students’  responses  indicate  substantial  increases  in  self-efficacy 
for  adaptive  performance  from  the  beginning  to  the  end  of  the 
course.  We  used  paired  crests  to  compare  pretraining  and  posttraining 
ratings.  Figure  3.7  shows  average  scores.  On  average,  students  reported 
“agree  somewhat”  before  training  and  “agree”  responses  after  training. 
The  increase  in  self-efficacy  was  statistically  signihcant  at  ^  <  0.001 
for  all  dimensions  and  for  leader  practice.  These  results  are  consistent 
with  Roselle,  2013,  which  compared  pretraining  and  posttraining  self- 
report  for  adaptability  behaviors  and  preferences  among  AWALP  stu¬ 
dents  with  a  control  group  that  did  not  participate  in  the  course. 
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Figure  3.7 

Self-Efficacy  Ratings  Pre-  and  Posttraining 
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The  results  in  Figure  3.7  are  presented  in  decreasing  order  of  effect 
size,  which  indicates  the  practical  significance  of  the  findings.'’  Effect 
sizes  for  the  differences  between  pre-  and  posttraining  ratings  were 
large  for  thinking  creatively;  ambiguous  situations;  and  learning  tasks, 
technologies,  and  procedures  and  were  moderate  for  the  remaining 
constructs. 

As  discussed  in  the  method  section  above,  the  questions  about 
self-efficacy  assess  improvement  from  the  course  indirectly  by  asking 
students  to  rate  their  skills  before  and  after  the  course  and  then  ana- 


Effect  sizes,  not  shown  in  Figure  3.7,  estimate  the  strength  of  the  relationship  between 
variables,  in  contrast  to  significance  levels,  which  reflect  the  probability  that  the  observed 
relationship  could  have  occurred  by  chance.  When  sample  sizes  are  large,  a  statistical  test  can 
be  significant,  even  if  the  size  of  the  test  statistic  (e.g.,  a  correlation  of  0.10)  is  too  small  to 
be  considered  practically  important.  For  f-tests,  Cohen,  1988,  describes  effect  sizes  of  0.8  as 
large,  0.5  as  moderate,  and  0.2  as  small.  For  changes  in  self-efficacy,  effect  sizes  {d)  were  0.78 
for  creative  thinking;  0.69  for  learning  tasks,  technologies,  and  procedures;  0.64  for  dealing 
with  ambiguous  situations;  and  0.61  for  cultural  adaptability,  and  ranged  from  0.50  to  0.54 
for  the  remaining  constructs. 
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lyzing  the  difference,  while  controlling  for  student  characteristics.  The 
reaction  survey  also  included  items  representing  a  different  approach, 
one  that  directly  asked  students  to  rate  how  much  the  AWALP  con¬ 
tributed  to  learning  dimensions  of  adaptive  performance  and  related 
constructs.  Figure  3.8  shows  the  distribution  of  responses  (“strongly 
agree,”  “agree,”  etc.),  as  well  as  the  average  rating  of  each  item.  The 
right  vertical  axis  (“Average  rating”)  tracks  the  mean  scores  line. 

Results  for  both  self-efficacy  and  these  direct  responses  are  gener¬ 
ally  consistent.  For  example,  for  self-efficacy,  thinking  creatively  and 
dealing  with  ambiguity  have  larger  changes  than  do  cultural  adaptabil¬ 
ity  and  decisionmaking  under  stress;  similarly,  the  direct  measures  of 
creative  thinking  and  ambiguous  situations  have  higher  means  than  do 
cultural  adaptability  and  decisionmaking  under  stress.  However,  it  is 
also  important  to  note  that  the  average  ratings  for  the  direct  measures 
in  Figure  3.8  are  substantially  higher  than  the  average  posttraining 
ratings  of  self-efficacy  in  Figure  3.7.  This  suggests  that  our  approach  to 

Figure  3.8 

Direct  Questions  About  AWALP  Impact 


I  I  Agree  strongly  CZI  Disagree  somewhat  *  Mean  scores 

I  I  Agree  H  Disagree 

I  I  Agree  somewhat  H  Disagree  strongly 
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measuring  improvement  indirectly  was  successful  at  avoiding  ceiling 
effects,  even  in  this  very  popular  course. 

Interest  in  Adaptive  Performance 

Students’  responses  also  show  significantly  greater  interest  in 
being  adaptable  following  AWALP  (see  Figure  3.9).  As  with  the  find¬ 
ings  for  self-efficacy,  students  reported,  on  average,  “agree  somewhat” 
before  training  and  “agree”  responses  after  training.  Results  of  paired 
T-tests  showed  statistically  significant  increases  for  all  dimensions  and 
for  leader  practice.  Results  show  a  large  effect  size  for  ambiguous  situ¬ 
ations  and  moderate  to  small  effect  sizes  for  the  remaining  constructs.^ 


Figure  3.9 

Interest  Ratings  Pre-  and  Posttraining 
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^  The  effect  size  was  0.77  for  ambiguous  situations,  0.44  for  decisionmaking  under  stress, 
and  0.47  for  leader  practice.  The  size  ranged  from  0.27  to  0.37  for  the  remaining  constructs. 
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Explanatory  Variables 

Results  show  substantial  improvement  in  self-efficacy  and  interest, 
even  after  accounting  for  students’  individual  differences  that  are 
associated  with  adaptive  performance.  We  used  analysis  of  variance 
to  examine  the  association  of  individual  characteristics  with  changes 
in  self-efficacy  and  interest.  These  analyses  indicate  whether  individ¬ 
ual  differences  are  associated  with  training  outcomes  and  whether  the 
changes  in  outcomes  appear  to  he  the  result  of  the  course  or  of  student 
characteristics  (or  of  both).  We  analyzed  Big  Five  personality  traits, 
including  openness  to  experience,  conscientiousness,  and  extraversion, 
as  well  as  other  dispositional  and  demographic  characteristics,  includ¬ 
ing  learning  goal  orientation  (mastery  and  performance  orientations), 
motivation  for  training,  and  time  in  service.  We  eliminated  some  vari¬ 
ables,  including  age,  which  was  highly  correlated  with  time  in  service; 
achievement  orientation,  which  was  highly  correlated  with  conscien¬ 
tiousness;  and  agreeableness,  which  was  not  internally  consistent.® 

Results  show  that  the  increase  in  both  self-efficacy  and  interest 
remain  statistically  significant  and  large  for  all  dimensions  of  adaptive 
performance  and  leader  practice  when  controlling  for  students’  individ¬ 
ual  characteristics.  There  were  main  effects  for  some  of  the  individual 
characteristics,  particularly  mastery  learning  goal  orientation,  extraver¬ 
sion,  and  time  in  service  for  both  self-efficacy  and  interest  outcomes.  In 
most  cases,  we  found  evidence  that  students  with  high  scores  on  these 
traits  or  with  longer  tenure  in  the  Army  reported  higher  self-efficacy 
and  interest  for  several  of  the  outcomes  when  measured  at  pretraining 
but  not  at  posttraining.  There  were  also  a  small  number  of  interactions 
between  time  (i.e.,  from  pretest  to  posttest)  and  between  some  of  the 
individual  differences  characteristics,  indicating  that  the  increase  in 
self-efficacy  and  interest  for  some  dimensions  of  adaptive  performance 
depended  on  student  characteristics,  but  the  interactions  were  few  and 
not  systematic  (see  Appendix  C). 

In  addition  to  examining  the  association  of  individual  character¬ 
istics  with  changes  in  attitudes,  we  conducted  hierarchical  regression 


®  Coefficient  a  for  agreeableness  =  0.68.  Coefficient  a  for  the  other  Big  Five  scales  ranged 
from  0.77  to  0.82. 
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analyses  to  assess  whether  individual  characteristics  could  account  for 
self-efficacy  and  interest  in  the  outcomes  measured  at  the  end  of  train¬ 
ing,  over  and  above  the  pretraining  measures  of  self-efficacy  and  inter¬ 
est.  The  increase  in  variance  explained  for  self-efficacy  measures  was 
statistically  significant  for  only  three  of  the  outcomes:  creative  think¬ 
ing;  learning  tasks,  technologies,  and  procedures;  and  decisionmaking 
under  stress.  For  these  outcomes,  the  increase  in  variance  explained 
was  largely  due  to  conscientiousness  and,  in  the  case  of  decisionmaking 
under  stress,  hy  time  in  service,  although  time  in  service  was  negatively 
associated  with  self-efficacy  for  decisionmaking  under  stress  at  the  end 
of  training. 

Overall,  the  results  indicate  that  training  outcomes  improved 
substantially,  regardless  of  individual  differences.  This  suggests  that 
measurement  of  individual  differences  is  not  needed  to  demonstrate 
the  benefits  of  AWALP  for  changes  in  learners’  attitudes  toward  adap¬ 
tive  performance,  and  we  would  not  recommend  selecting  participants 
for  AWALP  based  on  individual  differences  or  revising  the  course 
to  account  for  these  characteristics.  However,  because  we  measured 
learner  attitudes  rather  than  performance,  these  results  may  depart 
from  findings  of  other  studies  that  show  consistent  associations  of  char¬ 
acteristics,  such  as  cognitive  ability,  conscientiousness,  and  extraversion 
with  performance.  Therefore,  even  though  individual  characteristics 
did  not  contribute  to  the  variance  explained  for  most  of  the  posttrain¬ 
ing  outcomes,  it  is  worthwhile  to  continue  to  measure  individual  dif¬ 
ferences  if  the  goal  is  to  assess  transfer  of  training,  i.e.,  the  impact 
of  AWALP  (or  adaptability  training  more  broadly)  on  subsequent  job 
performance  when  prior  research  shows  that  individual  differences  do 
matter.  We  address  this  in  more  detail  when  discussing  suggestions  for 
future  research  in  Chapter  Six. 

Need  for  Adaptive  Performance 

In  addition  to  changes  in  students’  perceptions  about  their  capa¬ 
bilities  and  interests,  students’  views  of  their  job  requirements 
changed  following  AWALP.  The  next  five  figures  show  results  for  the 
perceived  need  for  adaptive  performance.  In  each  figure,  the  stacked 
bar  on  the  left  of  each  pair  shows  the  percentages  of  responses  in  each 
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category  (daily,  monthly,  etc.)  before  training,  indicating  how  much 
students  report  currently  engaging  in  the  behavior.  The  stacked  bar  on 
the  right  of  each  pair  shows  comparable  responses  after  training,  indi¬ 
cating  how  often  students  think  they  should  be  engaging  in  the  behav¬ 
ior.  Wilcoxon  Signed  Rank  tests  were  used  to  compare  the  differences 
in  responses.  Appendix  D  shows  the  degree  of  individual  change  in 
responses  (percentage  of  students  whose  pre-post  responses  did  not 
change,  changed  by  one  response  category,  by  two  response  categories, 
etc.).  Note  that,  while  we  had  multiple  items  for  each  dimension,  the 
hgures  show  results  for  only  one  item.  Unless  noted,  the  items  measur¬ 
ing  each  dimension  of  adaptability  not  presented  below  showed  similar 
patterns  of  results. 

Results  suggest  increased  need  for  all  dimensions  of  adaptive 
performance  in  students’  current  jobs.  These  changes  can  be  seen  by 
the  increasing  darker  gray  and  black  segments  in  each  pair  of  stacked 
bars  in  the  figures.  Dimensions  showing  the  greatest  shift  are  shown  in 
Figure  3.10;  those  with  a  smaller  but  still  statistically  significant  shift 
are  shown  in  Figure  3.11.  For  example,  in  Figure  3.10,  responses  to  a 

Figure  3.10 

Need  for  Creative  Thinking,  Learning,  and  Cultural  Adaptability  in  Current 
Job 
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Figure  3.11 

Need  for  Dealing  with  Ambiguity,  Interpersonal  Adaptability,  and 
Decisionmaking  Under  Stress  in  Current  Job 
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question  about  creativity  (i.e.,  “solve  problems  for  which  there  are  no 
easy  or  straightforward  answers”)  before  AWALP  show  that  41  percent 
of  students  reported  engaging  in  this  behavior  on  a  weekly  basis;  after 
AWALP,  52  percent  reported  that  they  should  engage  in  this  behavior 
on  a  weekly  basis.  Likewise,  13  percent  reported  engaging  in  this  aspect 
of  creative  thinking  on  a  daily  basis  before  AWALP,  and  27  percent 
reported  that  they  should  engage  in  this  behavior  daily  after  AWALP. 
The  degree  of  the  differences  between  the  beginning  and  the  end  of 
the  course  shows,  for  example,  that  16  percent  of  students  changed 
their  response  by  two  or  more  categories  in  a  positive  direction  (e.g., 
from  “monthly”  to  “daily”  or  “from  a  few  times  per  year”  to  “weekly”), 
32  percent  changed  their  response  by  one  category,  39  percent  did  not 
change  their  response,  and  so  on  (see  Appendix  D). 

Note,  however,  that  for  cultural  adaptability,  the  effects  are  dif¬ 
ferent  depending  on  the  item.  For  “Take  action  to  learn  about  social 
groups  other  than  my  own” — the  item  shown  in  Figure  3.10 — the 
effect  was  much  stronger  than  for  items  addressing  national  culture. 
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such  as  “Develop  relationships  with  people  from  different  countries” — 
the  item  shown  in  Figure  3.12.  These  differences  are  not  surprising 
given  that  the  focus  on  cultural  adaptability  in  AWALP  is  fairly  hroad 
and  is  not  specifically  on  national  culture. 

Results  also  suggest  an  increase  in  the  perceived  need  for 
engaging  in  most  of  the  behaviors  associated  with  leader  practice. 
These  include  seeking  consensus  from  subordinates  on  complex  deci¬ 
sions,  seeking  input  from  subordinates  on  complex  decisions,  train¬ 
ing  others  to  solve  complex  problems,  delegating  tasks  to  others  on 
which  they  might  fail,  and  conducting  AARs  (Figure  3.13).  Shifts  for 
two  of  these  behaviors — seeking  subordinate  consensus  and  conduct¬ 
ing  AARs — are  particularly  dramatic.  Prior  to  training,  44  percent  of 
students  reported  that  they  sought  consensus  from  subordinates  on  a 
weekly  or  daily  basis;  after  training,  79  percent  report  the  need  to  do 
so  weekly  or  daily.  Likewise,  46  percent  of  students  reported  that  they 
conduct  AARs  weekly  or  daily;  following  AWALP,  71  percent  reported 
that  they  should  conduct  AARs  weekly  or  daily.  This  shift  in  responses 
for  AARs  is  consistent  with  the  students’  responses  shown  in  Figure  3.2 
about  the  positive  impact  of  AWALP  on  AARs. 

Figure  3.12 

Need  for  Cultural  Adaptability  (People  from  Different  Countries)  in  Current 
Job 
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Figure  3.13 

Significant  Shifts  in  Need  for  Leader  Behaviors  in  Current  Job 
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In  contrast,  the  perceived  need  for  two  other  behaviors  associated 
with  leader  practice  in  the  current  joh  did  not  significantly  change  after 
AWALP.  The  distributions  of  responses  showed  that  students  already 
coach  subordinates  and  train  others  to  work  in  collaborative  teams 
(Figure  3.14). 

Interpreting  differences  between  “currently  engage”  and  “should 
engage”  is  somewhat  equivocal  because  we  did  not  ask  the  question  in 
the  same  way  in  both  surveys.  If  we  had  asked  participants  how  often 
they  should  engage  in  the  behavior  in  the  pretraining  survey,  their  rat¬ 
ings  might  have  been  higher,  resulting  in  potentially  smaller  differ¬ 
ences.  In  fact,  we  revised  the  surveys  for  ongoing  evaluation  of  the 
course  so  that  both  the  pre-  and  posttraining  surveys  ask  participants 
how  often  they  “should”  engage  in  the  behavior.  Use  of  the  revised 
surveys  by  AWG  in  a  subsequent  AWALP  class  showed  perceptions  of 
increased  need  for  all  the  dimensions  of  adaptability  and  most  aspects 
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Figure  3.14 

Nonsignificant  Shifts  in  Need  for  Leader  Behaviors  in  Current  Job 
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of  leader  practice,  with  the  magnitude  of  changes  from  pretraining  to 
posttraining  similar  to  the  results  reported  here. 


Conclusions 

This  chapter  provides  strong  evidence  of  student  satisfaction  with 
AWALP  in  terms  of  course  content,  structure,  and  delivery.  All  stu¬ 
dents  reported  that  they  would  recommend  the  course  to  others. 

Results  also  document  changes  in  learners  with  respect  to  affec¬ 
tive  training  outcomes  (i.e.,  attitudes)  as  a  result  of  taking  the  course. 
First,  responses  show  substantial  increases  in  self-efficacy  for  adapt¬ 
ability,  indicating  that  students  felt  more  capable  in  all  dimensions  of 
adaptive  performance  and  in  putting  adaptability  concepts  into  leader 
practice.  Second,  students  were  significantly  more  interested  in  work¬ 
ing  in  situations  that  require  adaptability.  Moreover,  improvements 
in  self-efficacy  and  interest  were  substantial  even  after  accounting  for 
individual  differences  in  characteristics  associated  with  adaptive  perfor¬ 
mance.  Third,  results  suggest  that  students  recognized  a  greater  need 
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for  engaging  in  most  of  the  adaptability  and  leader  behaviors  when 
returning  to  their  current  jobs. 

As  for  future  evaluation,  the  items  used  to  measure  attitudes 
addressed  a  series  of  general  situations  that  relate  to  the  dimensions  of 
adaptive  performance  and  the  leadership  of  others.  Therefore,  the  ques¬ 
tions  have  general  applicability  not  only  in  AWALP  but  also  in  other 
courses  that  train  for  adaptability. 


CHAPTER  FOUR 


Results:  Learning 


This  chapter  reports  on  measures  of  cognitive  and  behavioral  learning 
in  AWALP  based  on  knowledge  tests  of  course  concepts  and  observa¬ 
tions  of  adaptability  behaviors. 


Knowledge  of  Course  Concepts 

Method 

While  gaining  declarative  knowledge  is  not  a  main  course  goal,  some 
mastery  of  adaptability  concepts  is  a  prerequisite  for  understanding  the 
course  material  and  for  teaching  others  about  adaptive  performance. 
Using  course  materials,  and  in  collaboration  with  two  of  the  course 
guides,  we  developed  a  pool  of  35  multiple- choice  items  of  knowledge 
about  AWALP  concepts.  We  attempted  to  construct  a  test  with  a  range 
of  item  difficulty  levels  and  representing  the  breadth  of  topics  covered 
in  the  course.  For  example,  items  assessed  knowledge  of  the  definition 
of  adaptability,  how  dimensions  of  adaptive  performance  correspond 
to  definitions  or  descriptions  of  behaviors,  and  ways  to  develop  dimen¬ 
sions  of  adaptability  when  designing  training  or  leading  teams.  Each 
question  had  four  response  options. 

The  test  was  piloted  with  other  course  guides  and  several  Army 
Fellows  at  RAND  to  assess  whether  the  items  were  understandable  and 
at  an  appropriate  level  of  difficulty  and  how  long  it  took  to  complete 
the  test.  We  used  these  responses  to  revise  or  omit  questions,  resulting 
in  a  30-item  test  (for  example,  a  large  number  of  incorrect  responses  to 
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an  item  suggested  that  it  was  unclear,  did  not  reflect  material  taught  in 
the  course,  or  was  too  difficult). 

Two  items  that  many  students  answered  incorrectly  were  elimi¬ 
nated  after  the  test  was  given  because  further  examination  indicated 
that  the  questions  were  ambiguous  or  did  not  match  the  content 
addressed  in  the  course.  Thus,  scores  are  based  on  28  items.  The  test 
was  administered  on  day  0  and  again  on  day  10. 

Results 

Results  showed  increased  student  knowledge  of  AWALP  concepts. 

We  used  analysis  of  variance  to  analyze  the  change  in  pretraining  and 
posttraining  knowledge  test  scores,  controlling  for  general  cognitive 
ability  and  highest  level  of  education.  Average  scores  on  the  knowledge 
test  were  60  percent  correct  pretraining  {SD  =  12.50)  and  76  percent 
correct  posttraining  {SD  =  13.90).  The  increase  in  scores  was  statisti¬ 
cally  significant.  General  cognitive  ability  was  strongly  and  positively 
associated  with  scores  on  the  pretest  and  posttest,  but  there  was  no 
interaction  between  the  change  in  knowledge  test  scores  and  general 
cognitive  ability.'  Likewise,  highest  level  of  education  was  positively 
associated  with  scores  on  both  the  pretest  and  posttest,  but  there  was 
no  interaction  of  the  change  in  test  scores  with  level  of  education. ^ 
Thus,  student  knowledge  improved  regardless  of  general  cognitive 
ability  or  education  and  suggests  that  AWALP  was  successful  at 
fostering  knowledge  gain  for  a  wide  range  of  students.^ 


'  In  the  analysis  for  cognitive  ability,  for  the  change  in  pretraining  and  posttraining  knowl¬ 
edge  test  scores,  i^(l, 96)  =  139.82,  <  0.001,  and  the  effect  size  1.29,  which  is  a  very  large 
effect.  For  cognitive  ability,  /’(1,96)  =  17.53,/  <  0.001,  and  the  effect  sizes  (f)  were  0.41  for 
both  the  pretest  and  posttest,  which  is  a  moderate  to  large  effect.  For  the  interaction  of  test 
scores  and  cognitive  ability,  i^(l,96)  <  1,  not  significant. 

^  In  the  analysis  for  education,  for  the  change  in  pretraining  and  posttraining  knowledge 
test  scores,  i^(l,94)  =  96.95,/  <  0.001.  For  education,  /’(3,94)  =  4.57,/  <  0.01.  For  the  inter¬ 
action  of  test  scores  and  education,  /(1, 94)  <  1,  not  significant. 

^  Although  we  do  not  have  data  to  determine  how  representative  AWALP  students  are  of 
the  Army  at  large,  students’  scores  on  the  Wonderlic  Contemporary  Cognitive  Ability  Test 
are  generally  comparable  to  national  workforce  norms. 
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Figure  4.1  shows  the  distrihution  of  differences  scores  (i.e.,  posttest 
score  minus  pretest  scores).  Eighty-six  percent  of  the  students  showed 
increased  declarative  knowledge  (i.e.,  had  difference  scores  greater  than 
zero),  while  14  percent  performed  no  better  or  worse  on  the  posttest. 
The  average  difference  was  16  percent  (median  =  14  percent).  There 
were  no  obvious  patterns  of  responses  among  students  who  performed 
worse  (e.g.,  students  did  not  select  the  same  response  for  every  question, 
which  would  have  suggested  that  they  did  not  take  the  test  seriously).^ 


Peer  Evaluations 

Method 

In  the  first  of  the  three  classes  we  evaluated,  we  experimented  with 
peer  evaluations  as  a  behavioral  measure  of  individual  adaptive  perfor¬ 
mance.  We  designed  an  approach  based  on  the  peer  evaluation  proce- 

Figure  4.1 

Distribution  of  Change  Scores  on  Knowledge  Test 
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Removing  these  scores  increases  the  average  score  only  to  78  percent. 
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dures  used  at  the  U.S.  Military  Academy.  Our  procedure  asked  students 
to  anonymously  rate  three  of  their  peers  on  five  adaptive  performance 
dimensions  at  the  halfway  point  (phase  1)  and  at  the  end  of  the  course 
(phase  2).  In  an  effort  to  eliminate  uninformed  or  potentially  biased 
evaluations,  we  gave  each  rater  six  names  and  asked  them  to  select 
three  of  the  students  with  whom  they  had  worked  during  AWALP 
exercises  and  whom  they  did  not  know  before  coming  to  AWALP. 

The  constructs  covered  in  the  peer  review  included  how  much 
students  showed  flexibility  in  dealing  with  other  team  members,  had 
innovative  ideas,  interacted  effectively  with  others  who  have  different 
values  and  customs,  responded  easily  to  changes  in  the  situation,  and 
remained  calm  under  pressure.  There  was  one  item  for  each  construct, 
and  the  items  provided  examples  of  relevant  behaviors,  e.g.,  for  cre¬ 
ative  thinking,  examples  included  “Had  innovative  ideas  (including 
‘connected  the  dots’  and  presented  unique  approaches  or  courses  of 
action).”  As  elsewhere  in  the  evaluation,  six-point  scales  were  used, 
with  choices  from  “strongly  agree”  to  “strongly  disagree.”  In  addition, 
raters  had  an  option  to  indicate  “not  applicable”  for  exercises  they  felt 
did  not  call  for  the  behavior. 

Results 

Students  gave  high  ratings  of  adaptive  performance  to  their  peers, 
which  did  not  change  significantly  over  time.  On  average,  stu¬ 
dents  gave  their  peers  ratings  of  approximately  5  across  dimensions 
(Figure  4.2).  These  ratings  were  somewhat  higher  than  students’  self- 
efficacy  ratings  at  the  end  of  training  reported  earlier.  The  results  also 
indicate  that,  in  contrast  to  self-ratings,  adaptive  performance  did  not 
increase  between  the  midpoint  of  the  course  (phase  1)  and  at  the  end 
of  the  course  (phase  2).  In  fact,  while  generally  not  statistically  signifi¬ 
cant,  average  peer  ratings  decreased  somewhat  in  all  five  areas. 

While  the  results  suggest  that  adaptive  performance  did  not 
increase  between  the  two  phases,  there  are  alternative  plausible  explana¬ 
tions  for  the  peer  review  results,  making  interpretation  of  the  findings 
problematic.  One  difficulty  is  that  students  cannot  be  good  evaluators 
of  other  students  until  they  have  some  interactions  with  them  (which  is 
why  we  waited  until  day  5  for  the  first  measurement).  Thus,  significant 
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Figure  4.2 

Class  10  Peer  Evaluations  after  Phase  1  and  Phase  2 
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improvement  in  adaptive  performance  might  occur  in  the  early  part  of 
the  course  hut  would  go  undetected  hy  peer  evaluations.  Alternatively, 
ratings  may  reflect  social  desirability  bias,  and  ceiling  effects  limit  the 
possibility  of  higher  scores  over  time.  Another  issue  is  that  the  nature 
of  the  activities  differs  significantly  in  the  first  and  second  halves  of 
the  course,  making  an  evaluation  of  performance  during  each  period 
difficult  to  compare.  During  the  first  phase,  classroom  activities  are 
mixed,  with  relatively  straightforward  and  focused  field  exercises;  in 
the  second  phase,  students  are  asked  to  engage  in  much  more  complex 
field  problems.  A  third  issue  is  that  student  raters  are  themselves  learn¬ 
ing  during  the  course,  becoming  more  educated  and  discriminating 
and,  as  a  result,  may  be  better  critics  in  the  later  part  of  the  course. 
However,  this  explanation  seems  less  plausible  in  light  of  results  that 
will  be  presented  in  the  next  part  of  this  chapter. 

Associated  with  difficulties  of  interpretation  are  difficulties  with 
the  analysis.  The  biggest  issue  is  that  the  analysis  should  control  not 
only  for  the  characteristics  of  the  person  being  rated  but  also  for  the 
characteristics  of  the  individual  who  filled  out  the  instrument.  This 
requires  specific  statistical  expertise;  therefore,  the  analysis  may  not  be 
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practical  for  evaluating  adaptive  performance  on  a  routine  basis  in  a 
wide  range  of  courses  in  the  future. 

The  difficulty  of  interpreting  the  data,  and  difficulties  with  imple¬ 
menting  the  instruments  and  analyzing  the  data,  led  us  to  conclude 
that  peer  evaluations  will  not  be  a  good  approach  for  future  assess¬ 
ments  of  AWALP.  We  therefore  decided  to  eliminate  it  as  a  measure  for 
the  remaining  two  classes. 


Student  and  Guide  Ratings  of  Teams 

The  training  effectiveness  outcomes  we  have  reviewed  thus  far  address 
adaptive  performance  at  the  individual  level.  We  also  developed  a  mea¬ 
sure  to  assess  adaptability  at  the  team  level  for  practical  exercises,  using 
both  the  students  and  guides  as  raters.  The  measure  required  partici¬ 
pants  to  rate  whether  the  dimensions  of  adaptive  performance  were 
required  in  each  exercise  and  to  rate  the  team’s  effectiveness  on  these 
dimensions. 

We  sought  to  answer  the  following  questions  about  ratings  of 
adaptability  within  teams: 

•  Regarding  requirements,  do  students 

-  recognize  the  dimensions  of  adaptive  performance  tapped  in 
course  activities? 

-  become  more  accurate  in  their  judgments  over  time? 

•  Regarding  performance,  do  students  and  guides 

-  rate  teams  favorably  with  regard  to  behaviors  associated  with 
relevant  dimensions  of  adaptive  performance  in  the  exercises? 

-  agree  in  ratings  of  team  performance? 

-  improve  in  agreement  about  performance  ratings  over  time?^ 


^  We  also  intended  to  assess  whether  students  had  shared  mental  models  of  their  team’s 
performance  and  whether  agreement  in  students’  ratings  increased  over  time.  We  calculated 
intraclass  correlations  (ICCs)  for  team  ratings,  but  because  students  tended  to  have  limited 
variation  in  ratings,  with  most  ratings  ranging  from  4  to  6,  ICCs  appear  very  low.  Thus,  the 
ICCs  do  not  provide  a  good  indicator  of  within-group  agreement. 
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While  a  key  interest  is  whether  the  students  and  guides  generally 
agree  in  their  ratings  of  the  teams,  we  also  addressed  the  question  of 
whether  team  performance  improved  over  time  (from  day  3  to  day  7), 
particularly  based  on  data  from  the  course  guides  (we  might  expect 
generally  high  and  less-variahle  ratings  from  students,  as  we  observed 
with  peer  ratings  and  as  described  in  the  previous  section). 

Method 

We  developed  a  rating  form  consisting  of  15  items  addressing  six  of 
the  adaptive  performance  dimensions:  creative  thinking,  dealing  with 
ambiguity,  interpersonal  adaptability,  cultural  adaptability,  decision¬ 
making  under  stress,  and  physical  adaptability  (see  Table  4.1).  Most  of 
the  items  were  abbreviated  versions  of  the  items  used  to  assess  attitudes 


Table  4.1 

Dimensions  and  Items  for  Measures  of  Team  Adaptability 


Dimension 

Label 

Item 

Physical 

Phy 

Use  physical  strength  or  agility 

Interpersonal 

Inti 

Encourage  input  from  other  members 

Int2 

Modify  one's  behavior  to  get  along  with  others 

Int3 

Consider  others'  viewpoints 

Int4 

Read  others'  nonverbal  cues 

Ambiguity 

Ami 

Make  decisions  with  incomplete  information 

Am2 

Adjust  actions  rapidly  to  changes  in  the  situation 

Cultural 

Cull 

Interact  with  others  who  have  different  values  and 
customs 

Cul2 

Learn  rules  for  appropriate  interaction  with 
different  social  groups 

Creative  thinking 

Crel 

Look  at  problems  from  different  angles 

Cre2 

Develop  innovative  ideas 

Decisionmaking 
under  stress 

Strl 

Quickly  analyze  options 

Str2 

Remain  calm  under  pressure 

Str3 

Make  decisions  with  limited  time 

Planning 

Plan 

Plan  a  strategy  for  performing  the  task  in  advance 
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toward  adaptive  performance  reported  in  Chapter  Three.  Thus,  items 
were  only  indirectly  related  to  dimensions,  with  no  specific  mention 
of  the  dimension  itself.  For  example,  “Encouraging  input  from  other 
team  members”  related  to  the  interpersonal  dimension,  and  “Look¬ 
ing  at  problems  from  different  angles”  related  to  the  creative  thinking 
dimension.  We  did  not  include  items  assessing  learning  tasks,  tech¬ 
nologies,  and  procedures  because  we  expected  that  it  would  be  difficult 
to  observe  whether  others  engage  in  this  behavior.  At  the  recommenda¬ 
tion  of  the  guides,  we  also  included  an  item  assessing  planning  behav¬ 
ior,  given  its  importance  in  fostering  shared  mental  models.  With  the 
exception  of  physical  adaptability  and  planning,  which  had  one  item 
each,  there  were  two  to  four  items  for  each  dimension. 

Participants  were  asked  to  complete  the  rating  form  after  team 
exercises  on  three  days  of  the  course:  day  3  (“Firepoint”),  day  5  (“Star- 
burst,”),  and  day  7  (“Engage  the  Population”).  Like  other  practical  exer¬ 
cises  in  the  course,  these  exercises  involve  solving  problems  or  making 
judgments  for  which  teams,  on  average,  will  outperform  individuals 
(McGrath,  1984).  The  exercises  included  ambiguous  conditions  or 
information,  as  well  as  “curveballs” — unforeseen  events  or  conditions 
designed  to  induce  stress,  require  creative  thinking,  and  mandate  a 
change  in  plans.  One  of  the  important  lessons  from  several  of  the  tasks 
in  these  exercises  is  that  students  will  perform  more  effectively  if  they 
share  information  and  collaborate  than  if  they  attempt  to  complete  the 
tasks  alone.  The  “Firepoint”  exercise  involves  combat  casualty  care  and 
extraction.  Students  are  organized  into  eight-  to  nine-person  teams  and 
must  perform  a  variety  of  tasks  that  require  division  of  labor,  prob¬ 
lem  solving,  team  decisionmaking,  and  physical  strength  and  coordi¬ 
nation  that  exceed  the  capabilities  of  individual  members.  The  “Star- 
burst”  exercise  is  a  complex  capstone  exercise  for  the  first  half  of  the 
course  that  involves  a  counterinsurgency  environment.  Students  are 
organized  into  nine-  to  13 -person  teams.  The  exercise  starts  with  a  land 
navigation  event  requiring  a  variety  of  problem-solving  tasks.  Later, 
each  team  must  plan,  prepare,  and  execute  a  mission;  one  team  has  a 
patrolling  and  quick-reaction  force  mission;  one  has  a  personal  security 
detail  mission;  and  one  has  an  opposing  force  mission.  The  “Engage 
the  Population”  exercise  challenges  teams  to  deal  with  ambiguity  in 
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a  real  urban  environment.  Students  are  organized  into  eight-  to  nine- 
person  teams  and  are  instructed  to  collect  information  of  value  to  a 
commander  preparing  to  conduct  military  operations  in  the  city.  Stu¬ 
dents  are  told  to  consider  areas,  structures,  capabilities,  organizations, 
people,  and  events  (ASCOPE)  or  political,  military,  economic,  social, 
infrastructure,  information,  physical  environment,  and  time  (PMESII- 
PT)  variables,  but  otherwise,  instructions  on  the  commander’s  intent 
are  ambiguous.  At  the  end  of  the  exercise,  each  team  is  required  to  pre¬ 
pare  and  deliver  a  report  to  a  senior  military  officer. 

Students  were  instructed  to  write  a  code  at  the  top  of  the  form 
that  would  enable  us  to  match  their  responses  across  exercises  while 
maintaining  their  anonymity.  Then,  for  each  item  on  the  form,  stu¬ 
dents  were  asked  to  first  rate  whether  the  exercise  required  the  behav¬ 
ior  using  a  three-point  scale  (i.e.,  “not  at  all  or  limited  requirements,” 
“some  requirements,”  or  “substantial  requirements”).  Next,  they  were 
asked  to  rate  whether  their  team  as  a  whole  performed  effectively  on 
the  behavior  using  a  six-point  scale  ranging  from  “strongly  disagree” 
to  “strongly  agree.”  Students  completed  the  rating  form  independently. 

For  ratings  of  the  requirements  for  each  behavior,  the  guides  con¬ 
sulted  with  each  other  to  determine  one  set  of  judgments  that  consti¬ 
tuted  the  “right  answer.”  For  ratings  of  effectiveness,  guides  indepen¬ 
dently  rated  the  teams  that  they  accompanied  during  the  exercise. 

Results  for  Requirements 

Average  ratings  were  calculated  at  the  team  level.  Figures  4.3  through 
4.5  show  the  average  requirement  ratings  for  the  15  items  in  each  of  the 
three  exercises.  Although  the  items  were  presented  in  a  random  order 
on  the  survey  form,  they  are  grouped  by  adaptive  performance  dimen¬ 
sion  in  the  figures  to  facilitate  interpretation  of  results. 

The  similarity  between  students’  and  guides’  ratings  of 
requirements  for  adaptive  performance  generally  varied  across  the 
three  exercises.  We  considered  a  difference  between  students’  average 
ratings  and  the  “correct”  answer  of  0.5  or  greater  to  be  “large.”  As  shown 
in  Figures  4.3  and  4.4,  students  failed  to  recognize  that  some  adaptive 
performance  dimensions  were  not  relevant  to  the  Firepoint  and  Star- 
burst  exercises.  In  contrast,  as  shown  in  Figure  4.5,  students  recognized 
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Figure  4.3 

Student  and  Guide  Ratings  of  Firepoint  Requirements,  Day  3 


— Students 
— Guides 


NOTE:  Phy  =  physical,  Int  =  interpersonal.  Am  =  dealing  with  ambiguity,  Cul  =  cultural, 
Str  =  decisionmaking  under  stress.  Plan  =  planning. 
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Figure  4.4 

Student  and  Guide  Ratings  of  Starburst  Requirements,  Day  5 


*  Students 
— Guides 


that  Engage  the  Population  did  not  require  physical  adaptability,  sug¬ 
gesting  that  they  were  starting  to  discriminate  between  dimensions  of 
adaptive  performance  that  were  and  were  not  required.  However,  there 
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Figure  4.5 

Student  and  Guide  Ratings  of  Engage  the  Population  Requirements,  Day  7 


— Students 
— Guides 


were  also  large  discrepancies  between  student  and  guide  ratings  for  two 
or  more  other  items  in  each  exercise.'’ 

Students  appeared  to  become  more  accurate  in  their  ratings 
over  time.  As  shown  in  Figures  4.3  through  4.5,  the  number  of  items 
with  large  discrepancies  decreased  from  five  in  Firepoint  to  four  in 
Starburst  and  two  in  Engage  the  Population.  In  addition,  on  average, 
the  absolute  value  of  the  discrepancies  between  students  and  guide  rat¬ 
ings  decreased  over  time,  from  0.51  in  Firepoint  to  0.43  in  Starburst 
and  0.35  in  Engage  the  Population.  These  results  could,  however,  also 
reflect  a  general  lack  of  variability  in  students’  ratings,  coupled  with 
lower  variability  in  guides’  ratings  for  Starburst  and  Engage  the  Popu¬ 
lation  than  for  Firepoint. 


In  Firepoint,  these  included  the  two  items  about  dealing  with  ambiguity,  “Make  decisions 
with  incomplete  information,”  and  “Adjust  actions  rapidly  to  change  in  the  situation.”  In 
Starburst,  these  included  physical  adaptability  and  two  items  assessing  interpersonal  adapt¬ 
ability  (“Modify  one’s  behavior  to  get  along  with  others”  [Int2]  and  “Read  others’  nonverbal 
cues”  [Int4]).  In  Engage  the  Population,  these  included  one  of  the  same  interpersonal  adapt¬ 
ability  items  (Int2)  and  an  item  assessing  decisionmaking  under  stress  (“Make  decisions  with 
limited  time”  [Str3]). 
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Results  for  Performance 

Results  for  performance  are  shown  in  Figures  4.6  through  4.9.  Figure 

4.6  shows  the  results  aggregated  across  all  three  exercises,  and  Figures 

4.7  through  4.9  show  the  results  for  the  individual  exercises.  Note  that 
guides  were  told  not  to  rate  performance  for  items  with  “not  at  all 
or  limited  requirements”  ratings,  which  is  the  reason  for  gaps  in  the 
guides’  performance  ratings. 

In  general,  both  students  and  guides  gave  favorable  rat¬ 
ings  of  team  performance.  In  general,  students’  average  scores  hov¬ 
ered  around  5,  corresponding  to  an  “agree”  response,  and  most  of  the 
guides’  ratings  ranged  from  approximately  4  (i.e.,  “agree  somewhat”) 
to  5,  although  average  scores  were  helow  4,  toward  the  “somewhat  dis¬ 
agree”  range  for  a  few  of  the  items  (Figure  4.8). 

However,  students  consistently  rated  team  performance  more 
favorably  than  guides  did.  At  the  item  level  across  exercises,  discrep¬ 
ancies  between  guides’  and  students’  ratings  were  particularly  large  for 
three  of  the  items:  “Adjusting  actions  rapidly  to  changes  in  the  situa¬ 
tion”  (ambiguity  [Am2]  in  Figure  4.6)),  “Developing  innovative  ideas” 
(creative  thinking  [Cre2]),  and  “Quickly  analyzing  options”  (decision- 


Figure  4.6 

Student  and  Guide  Ratings  of  Team  Performance  Effectiveness  at  Item 
Level,  Aggregated  Across  Exercises 


•  Students 

*  Guides 
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Figure  4.7 

Student  and  Guide  Ratings  of  Team  Performance  on  Firepoint 


RAND  RR504-4.7 


*  Students 
— Guides 


Figure  4.8 

Student  and  Guide  Ratings  of  Team  Performance  on  Starburst 
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making  under  stress  [Strl]).  Differences  between  guides’  and  students’ 
ratings  also  were  statistically  significant  for  “Reading  others’  nonver¬ 
bal  cues”  (interpersonal  [Int4]),  “Looking  at  problems  from  different 
angles”  (creative  thinking  [Crel]),  “Making  decisions  with  limited 
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Figure  4.9 

Student  and  Guide  Ratings  of  Team  Performance  on  Engage  the  Population 
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time”  (decisionmaking  under  stress  [Str3]),  and  “Planning  a  strategy 
for  performing  the  task  in  advance”  (Plan). 

As  might  he  expected  given  the  differences  in  their  experience 
levels,  students  generally  discriminated  less  than  guides  did  in 
rating  team  effectiveness  across  the  different  dimensions  of  adap¬ 
tive  performance.  Figure  4.6  shows  that,  in  comparison  to  guides, 
students’  ratings  were  more  similar  across  the  items,  as  indicated  hy 
the  relatively  flatter  slope  for  student  ratings.  Similar  patterns  are  also 
evident  in  the  individual  exercises  shown  in  Figures  4.7  through  4.9. 
While  these  ratings  might  reflect  students’  true  opinions,  the  result 
could  indicate  a  tendency  not  to  think  too  much  about  the  questions 
before  responding,  a  possibility  that  is  supported  by  anecdotal  infor¬ 
mation  from  guides  about  how  quickly  students  sometimes  completed 
the  forms.  However,  it  is  interesting  to  note  that,  in  Engage  the  Popula¬ 
tion,  the  up-and-down  pattern  of  students’  ratings  more  closely  mirrors 
that  of  the  guides’  ratings  (Figure  4.9). 

At  the  same  time,  students  also  perceived  that  team  effectiveness 
was  increasing  by  the  third  exercise,  a  view  that  the  guides  did  not  nec¬ 
essarily  share.  Figure  4.10  depicts  the  average  effectiveness  scores  across 
the  items  in  each  of  the  three  exercises.  Note  the  relatively  flat  slope  for 
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Figure  4.10 

Student  and  Guide  Performance  Ratings  at  Exercise  Level  Aggregated 
Across  Items 


Firepoint  Starburst  Engage  the 

populace 


Exercise 
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guides’  ratings  (although  given  the  increasing  difficulty  of  the  exercises 
over  time,  the  relative  constancy  in  guides’  ratings  could  suggest  that 
teams  were  improving  in  adaptive  performance).  In  contrast,  the  stu¬ 
dents’  ratings  increased  from  the  hrst  to  the  third  exercise. 

Differences  between  students  and  guides  in  views  of  team  effec¬ 
tiveness  are  also  apparent  in  results  for  the  individual  exercises  shown 
in  Figures  4.7  through  4.9.  In  Firepoint  and  Starhurst,  there  is  some 
“crossover”  between  student  and  guide  ratings,  such  that  students  gave 
their  teams  higher  ratings  on  some  dimensions  and  guides  gave  teams 
higher  ratings  on  others.  However,  in  Engage  the  Population,  stu¬ 
dents  gave  their  teams  higher  ratings  than  guides  did  on  all  the  items. 
Thus,  students’  ratings  of  effectiveness  changed  over  time,  but  they 
appeared  to  become  more  confident  (or  overconfident)  about  the 
level  of  adaptability  in  their  teams  by  the  end  of  the  course. 

While  the  preceding  figures  show  results  for  the  three  AWALP 
classes  as  a  whole,  student  and  guide  ratings  also  can  be  calculated  for 
individual  teams.  For  example.  Figure  4.11  shows  differences  between 
guide  and  student  effectiveness  ratings  for  one  of  the  teams  in  Fire¬ 
point.  (Ratings  are  shown  only  for  required  behaviors  as  determined 
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Figure  4.11 

Student  and  Guide  Performance  Ratings  for  a  Single  Team,  Firepoint 


Item 
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by  the  guides.)  In  Chapter  Six,  we  discuss  ways  in  which  these  ratings 
could  be  used  to  support  AARs  in  the  course. 

Finally,  there  was  quite  a  bit  of  variability  among  some  of  the 
guides  in  rating  team  performance  effectiveness.  Some  guides  tended 
to  give  low  ratings  (Is  and  2s)  to  the  teams  that  they  observed,  while 
other  guides  gave  their  teams  higher  ratings  (from  3  to  6)  on  the  same 
criteria.  Guides  were  evaluating  different  teams,  so  it  is  possible  that 
the  lower  ratings  accurately  reflect  team  performance.  However,  given 
that  a  subset  of  the  guides  provided  the  lower  ratings,  it  is  also  possible 
that  there  are  systematic  differences  in  how  guides  are  assessing  per¬ 
formance,  which  in  turn  might  affect  feedback  to  the  teams.  In  Chap¬ 
ter  Six,  we  propose  a  strategy  to  assess  and  improve  consistency  among 
guides  in  evaluating  performance. 


Conclusions 

This  chapter  showed  clear  evidence  of  increased  cognitive  learning 
associated  with  AWALR  Results  showed  substantial  improvement  in 
declarative  knowledge  of  adaptive  performance  concepts  and  showed 
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that  learning  occurred  regardless  of  students’  general  cognitive  ability 
or  education.  Team  ratings  of  requirements  showed  that  students  were 
generally  accurate  in  identifying  the  adaptability  behaviors  required  in 
practical  exercises  and  that  the  accuracy  of  their  judgments  appeared 
to  improve  over  time. 

With  regard  to  ratings  of  performance,  both  the  guides  and  the 
students  agreed  that  teams  performed  at  least  somewhat  effectively 
across  most  adaptive  performance  dimensions,  even  as  exercises  became 
more  complex  and  demanding.  Guides’  ratings,  although  relatively 
constant  over  time,  may  indicate  behavioral  learning,  given  that  the 
exercises  were  increasing  in  difficulty.  However,  students  generally  dis¬ 
criminated  less  among  different  dimensions  of  adaptive  performance. 
Students  also  tended  to  inflate  ratings  of  team  effectiveness  relative  to 
guides’  ratings,  and  these  discrepancies  did  not  improve  over  time. 

This  chapter  also  featured  an  example  of  how  we  modified  our 
evaluation  approach  during  the  study.  Results  of  the  peer  evaluation 
showed  uniformly  high  ratings  given  to  peers  in  all  areas  of  adaptive 
performance  addressed  and  showed  limited  changes  over  time.  Because 
of  difficulties  of  interpretation,  implementation,  and  analysis,  we  dis¬ 
continued  use  of  peer  evaluations  after  the  first  class,  and  we  recom¬ 
mend  forgoing  use  of  peer  evaluations  in  further  evaluations  of  AWALR 
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Results:  Application  of  AWALP  Principles  on  the 
Job  and  Longer-Term  Attitudes  Toward  AWALP 


This  chapter  addresses  payoffs  of  AWALP  to  the  Army  in  terms  of 
graduates’  application  of  adaptive  performance  principles  on  the  job. 
We  conducted  telephone  interviews  with  students  and  their  supervisors 
to  assess  the  longer  term  impact  of  AWALP  on  adaptive  performance 
and  attitudes  after  students  returned  to  their  units. 


Purpose  of  Interviews 

We  interviewed  both  AWALP  graduates  and  their  supervisors  three 
months  and  six  months  after  the  course  ended.  There  were  two  pri¬ 
mary  goals.  The  first  goal  was  to  contribute  to  an  understanding  of 
how  AWALP  affects  participants’  work  when  they  return  to  their  units. 
The  central  question  was  whether  the  graduates  have  changed  profes¬ 
sionally  as  a  result  of  the  AWALP  training,  We  addressed  changes  in 
professional  conduct  through  questions  about  mentoring  and  training 
subordinates,  conducting  AARs  and  briefings,  delegating  tasks,  and 
seeking  input  from  others.  The  second  goal  was  to  understand  whether 
participants’  attitudes  about  the  course  changed  once  they  returned  to 
their  units.  We  addressed  attitudes  through  questions  about  recom¬ 
mending  AWALP  to  others,  needed  course  changes,  and  challenges 
to  implementing  AWALP  principles.  We  also  interviewed  graduates’ 
supervisors  to  learn  their  perspectives  on  how  AWALP  participants 
might  have  changed  professionally  as  a  result  of  the  AWALP  training. 
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Application  of  AWALP  Principles  and  Attitudes  Three 
Months  Post-AWALP 

Method  and  Response  Rates 

At  the  end  of  the  Class  10  AWALP  training  in  January  2013  and  at 
the  end  of  the  Class  11  training  in  April  2013,  we  asked  the  gradu¬ 
ates  whether,  in  three  months,  they  would  he  willing  to  participate  in 
a  brief  telephone  interview  about  their  AWALP  training  experience. 
Of  the  29  graduates  in  Class  10  (excluding  operational  advisors),  23 
(64  percent)  indicated  they  were  willing  to  be  interviewed.'  Of  the  35 
graduates  in  Class  11,  17  (49  percent)  indicated  that  they  were  will¬ 
ing  to  be  interviewed.  We  first  tried  to  contact  graduates  by  email  to 
request  telephone  interviews.  However,  because  we  had  initially  low 
responses  from  graduates  from  both  classes,  we  turned  to  phone  calls, 
which  resulted  in  much  greater  success.  From  early  April  through  early 
May  2013,  we  were  able  to  contact  and  interview  13  graduates  from 
Class  10.  In  August  2013,  we  were  able  to  contact  and  interview  11 
graduates  from  Class  11.  Thus,  in  total,  we  were  able  to  interview  24 
out  of  62  potentially  willing  participants,  or  39  percent,  across  classes. 
Table  5.1  shows  the  response  rates  by  class  and  overall.  The  24  gradu¬ 
ates  we  interviewed  represented  the  full  range  of  ranks  in  the  courses: 
8  percent  were  E5;  33  percent  were  E6;  25  percent  were  E7;  17  percent 
were  E8-9;  and  the  remaining  17  percent  were  commissioned  officers. 


Table  5.1 

AWALP  Graduate  Interview  Response  Rates,  by  Class 


Class  10 

Class  11 

Overall 

Total  in  class 

27® 

35 

62 

Willing  to  be  interviewed 

23 

17 

40 

Total  interviewed 

13 

11 

24 

Response  rate  (%) 

48 

31 

39 

®  Excludes  the  nine  operational  advisors  who  had  taken  AWALP  before. 


'  As  noted  in  Chapter  Two,  nine  students  were  taking  the  course  a  second  time  as  part  of 
their  operational  advisor  training.  We  excluded  these  students  from  our  analysis  of  applica¬ 
tion  of  course  principles  on  the  job. 
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The  three-month  follow-up  interviews  took  10  to  20  minutes 
each.  Each  followed  a  semistructured  interview  protocol.  We  asked 
interviewees  about  whether  they  altered  the  way  they  engaged  in  a 
number  of  adaptability  behaviors  reflecting  the  “leader  practice”  items 
reported  in  Chapter  Three  (see  Table  5.2),  with  prompts  on  various 
topics  if  interviewees  did  not  address  them.  If  interviewees  reported 
having  changed  their  behaviors,  we  asked  for  examples.  We  also  asked 
interviewees  who  in  the  organization  was  most  affected  by  the  changes. 
Finally,  we  asked  about  graduates’  attitudes  in  three  areas:  whether 
they  (1)  would  recommend  the  AWALP  to  others,  (2)  saw  the  need  for 
changes  in  the  course,  and  (3)  perceived  any  obstacles  in  implement¬ 
ing  the  principles  they  learned.  Appendix  D  provides  the  interview 
questions. 

Results:  Report  of  Changes  in  Work  Activities 

Eighty-eight  percent  of  graduates  reported  that  they  changed  pro¬ 
fessionally  as  a  result  of  AWALP.  We  asked  graduates  specifically 
about  whether  they  had  changed  the  way  they  coach,  train,  give  AARs, 
brief  commanding  officers,  delegate,  and  seek  subordinate  input. 
Table  5.2  shows  the  number  of  interviewees  who  responded  by  activ¬ 
ity.  Note  that,  because  of  interview  time  constraints  or  interviewee  job 


Table  5.2 

Frequency  of  Changes  in  Work  Activity  Three 
Months  After  AWALP  Graduation 


Activity 

Number  of 
Graduates 
Reporting 
Changes 

Coaching 

16 

Training 

15 

Delegating 

15 

Seeking  and  using  subordinate  input 

15 

Conducting  AARs 

10 

Briefing  commanding  officers 

3 
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functions,  we  did  not  ask  every  graduate  about  every  activity.  There¬ 
fore,  these  numbers  should  be  understood  as  only  rough  estimates  of 
the  total  of  each  kind  of  change. 

We  discuss  the  changes  reported  for  each  activity  more  specifi¬ 
cally  in  the  following  subsections,  starting  with  those  activities  that 
were  mentioned  most  often. 

Coaching 

The  majority  of  graduates  provided  specific  examples  to  demon¬ 
strate  how  they  have  modified  the  way  they  coach  and  mentor  sub¬ 
ordinates.  A  common  theme  was  that  post-AWALP,  graduates  realized 
the  value  of  spending  more  time  on  thoughtful  conversation  with,  and 
listening  to,  their  soldiers  and  on  how  such  processes  can  foster  better 
solutions  to  problems.  One  graduate  described  how  AWALP  made  him 
feel  more  empowered  and  how  he  wanted  to  pass  that  feeling  on  to  his 
subordinates  by  increasing  how  much  he  back-briefs  his  soldiers  and 
discusses  intent,  pitfalls,  and  choices  about  what  to  change  for  next 
time.  Another  graduate  explained: 

I  used  to  say  “this  is  how  you’re  going  to  learn  this,  OK,  no  ques¬ 
tions,  good,”  but  now  I  take  time  with  the  training,  have  my  sol¬ 
diers  ask  me  questions,  and  before  I  know  it,  they  know  how  to 
use  and  operate  the  machinery. 

Other  graduates  report  “listening  to  complaints  rather  than  blow¬ 
ing  them  off,”  “emphasizing  why”  to  approach  tasks  in  certain  ways, 
and  being  more  patient  and  encouraging  subordinates  to  “figure  things 
out  rather  than  just  doing  it  this  way.” 

Training 

The  majority  of  graduates  provided  specific  examples  of  how 
AWALP  has  affected  the  way  they  themselves  conduct  training 
within  their  units.  Most  frequently,  graduates  mentioned  ways  they 
have  been  able  to  use  what  they  learned  during  AWALP  to  improve 
how  they  conduct  physical  training  (PT)  and  range  work.  For  exam¬ 
ple,  two  graduates  (from  the  same  unit)  used  AWALP  concepts  to 
create  and  implement  a  PT  workout  that  included  land  navigation. 
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counter-improvised  explosive  device,  other  team  exercises,  and  getting 
through  obstacles.  Two  other  graduates  had  their  respective  soldiers 
work  in  teams  as  part  of  land  navigation  exercises  and  said  that  what 
they  learned  in  AWALP  helped  them  develop  and  manage  the  exer¬ 
cises.  Likewise,  Roselle,  2013,  found  that  even  one  month  after  gradu¬ 
ation,  AWALP  participants  had  begun  to  incorporate  adaptive  perfor¬ 
mance  principles  into  existing  training. 

Several  graduates  have  also  been  very  pleased  with  the  results  of 
following  the  AWALP  method  of  range  instruction,  which  empow¬ 
ers  participants  to  analyze  their  own  marksmanship  data  and  make 
decisions  on  how  to  alter  their  approach.  For  example,  one  graduate 
explained  that,  post-AWALP,  he  did  primary  military  instruction  “for 
soldiers  who  normally  wouldn’t  do  so  well  and  they  all  scored  in  the 
30s.”  Another  says  that  he  has  been  following  AWALP  shooting  proce¬ 
dures,  teaching  his  soldiers  exactly  what  AWALP  taught  him,  and  his 
soldiers  “have  absolutely  improved  their  gunnery  scores.” 

Most  graduates  reported  that  they  had  not  had  much  opportunity 
to  integrate  AWALP  principles  into  how  they  conduct  other  types  of 
training,  including  training  intended  to  promote  effective  teamwork, 
or  to  improve  complex  problem  solving  or  technical  skills  and  proce¬ 
dures  (other  than  shooting).  There  were,  however,  a  few  exceptions.  A 
few  graduates  reported  using  AWALP  principles  to  help  their  soldiers 
problem-solve,  work  more  effectively  in  teams,  and  prepare  for  ambig¬ 
uous  situations.  One  graduate  responded  that,  pre-AWALP,  in  training 
on  “Skill  Level  1,  ambush-type  raids,”  he  would  typically  repeat  the 
same  topics  because  he  thought  that  soldiers  learn  best  through  repeti¬ 
tion.  Based  on  what  he  learned  in  AWALP,  however,  he  is  now  trying 
to  incorporate  more  variety  and  ambiguity  into  this  type  of  training. 

A  few  graduates  explained  at  three  months  posttraining  that  they 
had  not  yet  had  the  opportunity  to  integrate  AWALP  principles  into 
training  but  have  thought  about  how  they  will  when  they  have  the 
opportunity.  One  graduate  had  plans  to  use  one  of  the  AWALP  exer¬ 
cises  a  few  months  down  the  road.  He  explained  that,  when  he  has  the 
opportunity,  he  will  break  his  soldiers  into  teams  (keeping  like  ranks 
together  so  that  privates  can  lead  their  peers)  and  present  surprise  chal¬ 
lenges  to  each  team. 


72  Innovative  Leader  Development 


Three  graduates  who  reported  not  integrating  AWALP  principles 
into  how  they  conduct  training  and  who  have  no  plans  to  do  so  in 
the  future  explained  that  they  are  not  currently  in  a  position  to  lead 
training. 

Delegating 

Most  graduates  responded  that  AWALP  changed  how  they  del¬ 
egate,  reflecting  an  outcomes-based  approach.  A  common  theme 
was  graduates’  conscious  intention  to  delegate  in  a  way  that  provided 
less  procedural  guidance  and  more  clarity  on  the  ultimate  goal.  One 
graduate  summarized  what  he  learned  from  AWALP  as  the  following: 
“My  way  might  be  the  way  I’d  do  it,  but  let’s  see  what  happens  with 
less  guidance.”  Another  graduate  explained  that,  post-AWALP,  his  del¬ 
egation  style  is  to  say,  “Here’s  your  mission,  you’ve  got  the  nuts  and 
bolts,  you  build  the  motor  the  way  you  think  it  needs  to  be  built,  then 
we’ll  look  at  it  and  tweak  it.”  Another  explained  that,  before  AWALP, 
if  the  subordinate  “didn’t  get  it  right  the  first  time,  [he  would]  just  do 
it  himself  .  .  .  But  AWALP  said,  don’t  give  them  the  answers,  make 
them  do  the  work.”  Thus,  since  AWALP,  he  has  been  following  that 
principle.  A  third  graduate  applied  AWALP  delegation  principles  to 
how  he  organizes  his  unit,  such  that  the  more  experienced  soldiers  now 
mentor  those  newer  to  the  unit.  Likewise,  Roselle,  2013,  also  identified 
a  theme  among  graduates  that  they  were  more  prone  to  empower  team 
members  to  find  their  own  solutions  to  issues  that  arose  in  training. 

Not  everyone  with  subordinates  completely  embraced  AWALP 
principles.  For  example,  one  graduate  seemed  to  continue  to  provide 
quite  strict  procedural  guidance  but  did  provide  more  clarity  on  the 
ultimate  goal,  explaining  that  he  now  says  to  his  subordinates  “do  it  in 
this  manner  and  this  is  why  I  want  you  to  do  it  in  that  manner.” 

Even  graduates  without  direct  managing  responsibility  for  subor¬ 
dinates  reported  that  they  were  applying  AWALP  delegation  principles 
to  their  work  in  some  way.  One  described  how  he  now  more  frequently 
looks  to  others’  expertise  to  help  with  project-based  work.  Another, 
describing  himself  as  a  “worker  bee”  with  little  opportunity  to  delegate, 
explained  that,  during  the  Advanced  Leader  Course,  he  did  very  well 
on  a  delegating  exercise  because  of  what  he  learned  in  AWALP. 
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Five  graduates  indicated  that  AWALP  did  not  change  their  del¬ 
egation  style.  Three  graduates  responded  that  they  had  no  opportunity 
to  delegate,  and  thus,  AWALP  had  no  impact  on  them  for  this  skill; 
two  said  that  they  already  delegated  per  AWALP  principles  before  the 
training. 

Seeking  and  Using  Subordinate  input 

Graduates’  responses  to  whether  their  AWALP  training  experience 
affected  how  they  seek  and  use  input  from  subordinates  varied  and 
can  be  understood  on  a  continuum.  The  majority  responded  that 
they  learned  in  AWALP  that  subordinates  can  have  good  ideas  and  that 
graduates  can  now  ask  subordinates  for  more  input.  Several  graduates 
explained  that  they  have  always  asked  for  input  from  subordinates,  but 
AWALP  strengthened  the  habit.  One  explained  that  he  always  asked 
subordinates  for  input;  however,  he  explained  that  AWALP  changed 
his  rationale  for  doing  so.  Specifically,  pre-AWALP,  he  realized  the  ben¬ 
efit  of  learning  from  substantive  experts  about  a  given  topic  area.  Post- 
AWALP,  he  began  asking  for  input  as  a  relationship-building  strategy. 
Roselle,  2013,  also  found  that  graduates  reported  encouraging  input 
from  team  members  to  a  greater  degree  after  completing  AWALP 
training. 

Finally,  three  graduates  explained  that  they  have  always  asked  for 
input  and  that,  in  this  regard,  AWALP  training  did  not  affect  their 
professional  work. 

Conducting  AARs 

Fewer  than  half  of  the  graduates  responded  that  AWALP  affected 
the  way  they  conduct  AARs.  However,  most  respondents  shared 
the  sentiment  that  the  AWALP  method  is  more  useful  than  the  tra¬ 
ditional  “three  improvement,  three  sustainment”  model  that  has 
been  the  Army  status  quo.  While  the  goal  of  AARs  is  to  understand 
what  happened  in  an  event,  why  it  happened,  and  how  it  could  be 
done  better,  sometimes  these  reviews  become  simplified  to  the  point 
that  they  become  little  more  than  short  lists  of  what  was  done  well  and 
what  needs  to  improve.  In  contrast,  in  AWALP,  an  AAR  is  a  signifi¬ 
cant  learning  process  that  encourages  critical  thinking,  reflection,  and 
discussion  among  all  those  who  participated  in  the  event.  Comments 
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from  graduates  showed  that  they  were  applying  this  approach  to  AARs 
on  the  joh.  For  example,  one  graduate  explained  that  he  now  asks  his 
subordinates,  “What  was  your  intent  today?  How  did  you  achieve  it? 
Tell  me  about  the  end  state.”  Several  others  mentioned  moving  away 
from  the  “three  up,  three  down”  model  to  asking  soldiers  to  talk  about 
what  they  did,  how  they  felt  they  did,  what  could  have  improved  the 
activity,  and  what  could  have  been  a  better  course  of  action. 

Few  graduates  mentioned  trying  to  influence  their  peers’  or  com¬ 
manders’  AAR  style.  However,  three  graduates  mentioned  an  intention 
to  share  principles  about  conducting  AARs  learned  in  AWALP  with 
others  in  their  units  in  addition  to  their  subordinates. 

For  the  most  part,  the  graduates  who  had  not  changed  the  way 
their  approach  to  AARs  explained  that  they  had  not  yet  had  the  oppor¬ 
tunity  to  conduct  AARs.  A  small  minority,  including  two  with  Spe¬ 
cial  Forces  backgrounds,  said  they  already  follow  AWALP  principles  in 
giving  AARs. 

Briefing  Commanding  Officers 

For  the  majority  of  graduates,  their  AWALP  training  did  not 
change  the  way  they  brief  commanding  officers.  Only  three  gradu¬ 
ates  provided  examples  of  how  the  AWALP  training  affected  how  they 
briefed  a  commanding  officer.  One  graduate  explained  that,  post- 
AWALP,  he  now  presents  proposed  solutions  to  problems  as  part  of  the 
briefing. 

The  remaining  graduates,  however,  responded  that  AWALP  did 
not  change  how  they  brief  a  commanding  officer  or  that  they  have  not 
had  an  opportunity  to  brief  at  all.  Four  graduates  said  that  AWALP  did 
not  affect  how  they  brief  commanding  officers  because  they  already 
follow  AWALP  principles  to  brief  One  explained  that  he  was  also 
required  to  brief  commanding  officers  in  “the  formal  way.” 

Results:  Use  of  Outcomes-Based  Strategies  for  Training,  Delegating, 
and  Mentoring 

While  graduates  did  not  say  so  directly,  some  of  the  examples  above 
demonstrate  the  use  of  outcomes-based  strategies  in  leading  others.  We 
also  reviewed  interview  transcripts  more  holistically  to  analyze  how 
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much  AWALP  seems  to  affect  graduates’  use  of  outcomes-based  strate¬ 
gies  more  broadly.  Of  the  24  three-month  interviewees,  11  clearly 
expressed  ideas  that  suggested  AWALP  influenced  their  use  of  out¬ 
comes-based  strategies.  Some  of  the  quotes  above  reflect  this  change. 
The  following  is  another  representative  quote: 

AWALP  has  impacted  everything  from  the  way  I  give  instruc¬ 
tions  at  the  beginning  of  training  through  the  AAR.  I’m  more 
concerned  about  outcomes  and  having  subordinates  think  about 
the  process  and  solutions.  I  now  encourage  more  thinking  rather 
than  the  old  way  of  telling  them  how  to  do  each  step. 

Eight  graduates  seemed  to  loosely  express  a  shift  toward  increased  use 
of  outcomes-based  strategies.  For  example, 

I  have  soldiers  think  more  for  themselves. 

Now  I  use  a  really  hands-off  approach,  and  if  they  get  off  track,  I 
try  to  coach  them  back. 

I  have  a  lot  of  newly  promoted  sergeants;  I’m  delegating  a  little 
more  to  the  new  guys,  having  the  old  guys  mentor  the  new  guys. 

Only  five  graduates  did  not  express  any  ideas  related  to  an  outcomes 
approach. 

Results:  Who  Is  Most  Affected  by  Graduates'  Changes 
Graduates’  responses  to  questions  about  coaching,  training,  del¬ 
egating,  and  seeking  input  demonstrate  that  nearly  all  AWALP 
graduates — 20  of  24  interviewed — feel  that  they  have  been  able  to 
integrate  AWALP  principles  into  their  work  with  subordinates  in 
some  way.  Direct  comments  about  subordinates’  reactions  to  AWALP 
principles  include,  “Subordinates  have  been  very  receptive”  and  “they 
feel  my  trust,  confidence  in  them.” 

Seven  graduates  reported  also  attempting  to  share  AWALP  prin¬ 
ciples  with  their  peers,  including  three  who  are  working  with  a  particu¬ 
larly  receptive  unit  that  has  sent  a  number  of  leaders  to  AWALP  train¬ 
ings.  For  example,  one  said  “I  am  trying  to  show  other  platoon  leaders 
how  well  AWAFP  principles  can  work  with  his  [sic]  platoon.” 
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Four  graduates  mentioned  that  they  have  also  been  able  to  have 
some  impact  on  the  unit  through  sharing  AWALP  principles  with 
their  commanders.  One  graduate  responded  that  command  climate 
and  leadership  has  been  comfortable  with,  and  supportive  of,  integrat¬ 
ing  AWALP  principles  into  unit  practices.  Another  noted  the  effect  on 
his  unit,  but  qualihed  his  response,  believing  AWALP  principles  have 
spread  “all  the  way  up  to  the  company  level  but  not  beyond  that.” 

Finally,  four  graduates’  comments  suggested  that  they  had  not 
successfully  spread  AWALP  principles  to  others  (subordinates,  peers, 
or  the  unit)  on  return  from  the  training.  One  respondent  explained 
that  he  is  the  “new  guy  in  the  unit,  so  it’s  hard  for  others  to  recognize 
changes  in  [his]  work  behavior.”  Another  said  that  no  one  recognizes 
any  changes  in  his  professional  work  because  of  the  nature  of  his  job. 

As  we  describe  below,  a  number  of  other  graduates  seem  to  be 
implementing  AWALP  principles  with  their  subordinates,  yet  they  also 
cited  command  climate  and  leadership  buy-in  as  challenges  to  fully 
implementing  AWALP  principles. 

Results:  Changes  in  Attitudes 

We  asked  three  questions  that  addressed  graduates’  attitudes  toward 
AWALP  three  months  after  training  and  compared  these  responses  with 
those  reported  at  the  end  of  AWALP.  These  comparisons  are  discussed 
below.  Overall,  answers  to  each  of  the  three  questions  remained 
relatively  constant  on  day  10  of  training  and  three  months  later. 

Recommending  AWALP  Training  to  Others 

Graduates’  attitudes  about  recommending  AWALP  training  to 
others  remained  highly  favorable  three  months  after  the  course. 

As  noted  in  Chapter  Three,  at  the  end  of  training,  all  students  reported 
that  they  would  recommend  AWALP  to  others.  Three  months  after  the 
training,  every  graduate  also  responded  that  he  would — and  in  many 
cases  has — recommended  AWALP  to  others.  One  said  that  he  has  been 
recommending  AWALP  ever  since  he  returned,  and  another  echoed 
that,  saying  he  tells  everyone  to  go  to  the  course  because  “it  will  change 
how  you  think  about  and  do  things.”  Another  said  that  AWALP  should 
be  required  when  soldiers  become  NCOs. 
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Recommendations  to  Improve  AWALP  Training 
The  nature  of  recommendations  for  improving  AWALP  training 
also  did  not  change  markedly  between  the  end  of  the  course  and 
the  three-month  interviews.  At  the  end  of  the  course,  more  than 
one-half  of  the  graduates  suggested  a  change,  and  three  months  after 
AWALP,  nine  of  24  graduates  suggested  a  change.  Fifteen  graduates 
said  they  had  no  recommendations  because  they  believe  that  all  aspects 
of  the  AWALP  training  should  stay  as  they  are.  Of  the  nine  graduates 
who  offered  a  recommendation  for  change,  only  one  made  a  critical 
comment;  he  thought  that  the  training  should  be  “tighter”  because  he 
felt  that  the  more  junior  attendees  were  “seeing  it  as  play  time.”  The 
other  eight  made  a  variety  of  more  positive  suggestions;  these  included 
that  more  people  should  have  to  go  through  the  AWALP  training, 
that  the  training  should  be  longer  in  terms  of  days,  that  more  range 
time  would  be  useful,  and  that  more  “chaos”  in  the  exercises  would  be 
useful.  Another  graduate  commented  on  the  usefulness  of  not  restrict¬ 
ing  AWALP  activities  according  to  rank.  He  explained  that  he  “got 
a  lot  out  of  interacting  with  more  senior  soldiers,”  such  that  now  he 
understands  majors  and  higher-level  officers  better,  which  helps  in 
learning  how  to  interact  with  them,  including  asking  for  permission  to 
implement  AWALP  principles. 

Challenges  to  Implementing  AWALP  Principles 

Similarly,  graduates  continued  to  report  command  climate  and 
leadership  as  the  chief  obstacles  to  implementing  AWALP  princi¬ 
ples.  The  most  commonly  mentioned  challenges  related  to  entrenched 
attitudes,  command  climate,  and  “Armyisms,”  such  as  “old-school 
sergeant  majors,”  and  the  “I-told-you-so  mind-set.”  One  graduate 
described  the  challenges  of  peers  who  are  “stuck  in  their  ways”  and  also 
of  contractors  at  range  control  who  limit  how  much  he  is  permitted  to 
alter  range  work  to  adapt  AWALP  strategies.  Several  other  interviewees 
expressed  concerns  about  the  time  and  resources  needed  to  implement 
AWALP  principles,  particularly  for  training. 

Twelve  of  the  graduates  mentioned  challenges  in  implementation. 
This  is  likely  an  underestimate  of  the  number  who  actually  saw  chal¬ 
lenges  because  we  modified  our  interview  protocol  between  the  Class 
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10  and  Class  11  interviews  to  ask  more  explicitly  about  challenges  in 
the  second  round.  Consequently,  Class  1 1  graduates  were  more  specific 
about  challenges  they  had  encountered  than  were  Class  10  graduates. 

Other  Feedback 

The  only  feedback  graduates  offered  beyond  the  structured  inter¬ 
view  protocol  was  their  praise  for  the  AWALP  training.  The  com¬ 
ments  they  made  included  “Excellent”;  “Great”;  “Outstanding”;  “One 
of  my  favorite  courses”;  “People  were  very  professional;  very  wise,  very 
engaged  teachers”;  and  “The  best  [engagement  skills  training]  I’ve  ever 
been  to,  best  ten  days  of  instruction  I’ve  ever  gotten  out  of  the  military, 
period — in  over  ten  years.”  Two  graduates  made  the  point  that  AWALP 
makes  young  leaders  more  three  dimensional  by  teaching  them  to  plan, 
teach,  and  execute. 

The  two  graduates  with  Special  Forces  backgrounds  mentioned 
that  AWALP  did  not  impact  the  way  they  work  in  any  of  the  areas 
above  because  they  had  already  learned  to  incorporate  adaptive  perfor¬ 
mance  principles  into  their  work  through  their  prior  training. 


Application  of  AWALP  Principles  and  Attitudes  Six 
Months  Post-AWALP 

Method 

We  followed  up  with  the  Class  10  graduates  roughly  six  months  after 
AWALP  training,  in  August  and  September  2013.  We  engaged  gradu¬ 
ates  in  brief,  five-  to  ten-minute  conversations  about  whether  they  had 
been  able  to  implement  AWALP  principles  to  any  greater  extent  since 
we  last  spoke  in  April  2013.  If  they  responded  affirmatively,  we  asked 
what  they  had  been  able  to  do.  If  they  responded  negatively,  we  asked 
why.  We  were  able  to  speak  with  ten  of  the  16  Class  10  graduates  (63 
percent). 

Results 

At  six  months  after  training,  more  than  half  the  graduates 
reported  that  they  were  continuing  to  apply  AWALP  principles  to 
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their  professional  performance,  particularly  in  how  they  delegate 
to,  and  solicit  advice  from,  subordinates  and  design  new  training. 

Interviews  suggested  that,  six  months  after  training,  implementation 
of  AWALP  principles  had  not  increased  from  three  months  after  the 
course.  However,  three  graduates  who  were  implementing  AWALP 
principles  at  the  three-month  period  indicated  that  they  had  changed 
units  and  roles  and  had  had  little  to  no  opportunity  to  implement 
AWALP  principles  in  their  new  positions.  Finally,  one  graduate  (who 
had  indicated  some  efforts  at  the  three-month  period)  reported  after  six 
months  that  finding  the  time  to  implement  and  actually  rememher  the 
AWALP  principles  had  heen  major  challenges. 


Interviews  of  Supervisors 

Method 

We  asked  the  Class  11  interviewees  {n=  11)  and  the  Class  10  interview¬ 
ees  with  whom  we  followed  up  at  six  months  after  AWALP  {n  =  10) 
whether  we  could  conduct  a  brief  telephone  interview  with  their  super¬ 
visors  about  ways  in  which  graduates  might  have  changed  profession¬ 
ally  as  a  result  of  the  AWALP  training.  Asking  graduates’  permission 
to  talk  with  supervisors  was  used  to  comply  with  Institutional  Review 
Board  requirements.  No  graduates  refused  to  give  us  permission  to  talk 
with  their  supervisors  or  were  outwardly  resistant  to  the  request.  How¬ 
ever,  we  were  successful  in  speaking  with  only  four  of  the  21  supervi¬ 
sors  we  sought  to  interview.  The  following  results  should  be  interpreted 
with  caution,  because  the  sample  size  was  extremely  small. ^ 


^  For  nine  of  the  17  not  interviewed,  we  ended  up  not  asking  for  contact  information. 
In  seven  cases,  all  six  months  after  AWALP,  the  graduate’s  supervisor  had  changed  since 
AWALP.  Thus,  the  current  supervisor  would  not  be  able  to  speak  to  a  change  in  the  gradu¬ 
ate’s  performance  from  pre-  to  post-AWALP.  In  another  case,  the  commander  was  overseas 
and  unreachable.  Finally,  we  decided  not  to  pursue  one  graduate  because  his  commander 
worked  for  AWG.  For  four  of  the  17  not  interviewed,  the  graduate  indicated  a  willingness  to 
provide  contact  information  in  the  future  or  have  the  supervisor  call  us,  but  follow-ups  were 
unsuccessful.  The  remaining  three  graduates  did  provide  supervisor  contact  information,  but 
we  were  unable  to  actually  get  in  touch  with  the  supervisor.  Some  of  these  numbers  may  not 
have  been  reliable  contact  information. 
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We  asked  supervisors  about  their  role  in  selecting  students  for 
AWALP,  the  criteria  they  used  for  selection,  and  how  the  graduates 
changed  after  completing  AWALP. 

Results 

Each  of  the  four  supervisors  we  spoke  with  responded  that  they 
saw  meaningful,  positive  changes  in  how  the  graduates  conducted 
themselves  professionally  on  returning  from  AWALP.  The  super¬ 
visors  were  most  able  to  comment  on  their  communication  with  the 
graduates  and  how  the  graduates  planned  training.  The  supervisors 
felt  that  graduates  were  more  self-reliant  and  better  at  communicat¬ 
ing  logical  thought  processes.  They  also  reported  that  graduates  were 
designing  training  that  would  help  develop  AWALP  principles  in  sub¬ 
ordinates  and  were  doing  so  in  creative  yet  viable  ways.  Supervisors 
were  generally  not  able  to  comment  on  how  the  graduates  interacted 
with  subordinates  in  terms  of  coaching,  delegating,  or  soliciting  input 
because  they  did  not  see  these  interactions.  One  supervisor  explained 
that,  for  AWALP  graduates  to  have  an  effect  on  their  units  posttrain¬ 
ing,  they  must  already  have  earned  the  “white  space” — or  respect — 
within  the  unit  prior  to  going  to  the  training. 


Conclusions 

Graduates  reported  substantial  application  of  AWALP  principles  on 
the  job  after  three  months,  especially  in  the  areas  of  coaching,  training, 
delegating  to  subordinates,  and  seeking  subordinate  input.  Moreover, 
many  graduates  reported  applying  outcomes-based  strategies  for  train¬ 
ing,  delegating,  and  mentoring,  leaving  more  freedom  for  subordi¬ 
nates  to  address  tasks  and  challenges.  For  most,  these  positive  effects  of 
AWALP  were  sustained  six  months  after  graduation.  Reported  changes 
were  less  common  in  other  areas,  such  as  conducting  AARs  and  brief¬ 
ing  commanding  officers,  although  many  of  the  graduates  reported  no 
opportunity  to  engage  in  these  activities  in  their  current  roles.  While 
AWALP  principles  were  successfully  disseminated  to  graduates’  subor¬ 
dinates,  dissemination  was  somewhat  less  successful  to  peers  and  com- 
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manders  and  throughout  the  unit.  Results  reflect  some  of  the  factors 
in  the  work  environment  that  inhibit  training  transfer,  such  as  insuf- 
flcient  opportunities  to  engage  in  newly  learned  skills  on  the  joh  and 
an  unsupportive  organizational  culture  (Grossman  and  Salas,  2011; 
Rouiller  and  Goldstein,  1993;  Salas  et  ah,  2012). 

Graduates  also  remained  extremely  positive  about  AWALP  three 
months  after  the  course.  All  graduates  would  still  recommend  the 
course  to  others,  and  few  recommended  course  changes.  However, 
graduates  also  saw  the  same  potential  obstacles  to  applying  AWALP 
principles  once  back  at  their  units,  with  command  climate  (“Army- 
isms”)  and  entrenched  leadership  still  the  most  frequently  mentioned 
challenges. 

Supervisors  were  likewise  positive  about  the  effect  of  AWALP  on 
graduates’  behavior  after  the  course,  particularly  with  respect  to  observ¬ 
ing  improvements  in  training  planning  and  communications  with  the 
commanding  officers.  However,  a  much  larger  sample  of  supervisors  is 
needed  to  draw  meaningful  conclusions  about  changes  in  graduates’ 
behavior.  Difficulties  in  recruiting  supervisors  likely  occurred,  in  part, 
because  the  evaluation  was  a  research  study  and  therefore  required 
graduates’  consent  to  contact  their  supervisors,  and  we  were  not  able  to 
reach  all  graduates.  This  requirement  would  not  be  in  effect  when  the 
Army  conducts  its  own  evaluations  in  the  future.  In  Ghapter  Six,  we 
discuss  additional  strategies  to  engage  supervisors  in  follow-up  discus¬ 
sions,  as  well  as  alternative  measures  of  training  transfer  and  impact. 


CHAPTER  SIX 


Summary,  Conclusions,  and  Recommendations 


This  chapter  summarizes  our  results  and  draws  conclusions  based  on 
the  findings.  We  then  present  recommendations  about  AWALP  instruc¬ 
tion  and  about  how  the  evaluation  of  AWALP  could  be  improved  in 
the  future.  We  then  broaden  our  discussion  outside  of  the  current 
course,  addressing  how  the  AWALP  approach  to  adaptability  training 
could  be  expanded  in  the  Army.  Finally,  we  describe  how  the  methods 
used  in  this  study  might  be  applied  to  evaluating  adaptability  training 
in  other  contexts. 


Summary  of  Key  Findings  and  Conclusions  from  AWALP 
Evaluation 

Table  6.1  summarizes  key  findings  from  the  evaluation  of  AWALP. 
Results  provide  evidence  of  course  success  across  a  range  of  measures. 
The  following  sections  discuss  these  findings  in  more  depth. 

Reactions 

Surveys  administered  at  the  end  of  AWALP  showed  that  students 
were  extremely  satisfied  with  the  course  structure,  content,  and 
delivery.  In  fact,  students’  reactions  toward  AWALP  were  much  more 
favorable  than  those  active-duty  leaders  expressed  about  institutional 
training  in  general  in  the  2013  CASAL  survey  (Riley  et  ah,  2013). 
AWALP  students  attribute  their  learning  largely  to  the  content  of  the 
course  and  to  the  training  methods,  which  differ  substantially  from 
typical  Army  training.  All  AWALP  students  would  recommend  the 
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Table  6.1 

Summary  of  Key  Findings 


Outcomes 

Key  Findings 

Reactions  to 

AWALP 

Students  were  extremely  satisfied  with  the  course  structure, 
content,  and  delivery,  attributing  learning  largely  to  course 
content  (adaptive  performance)  and  the  training  methods, 
which  differ  substantially  from  typical  Army  training. 

Attitudes 
toward  adaptive 
performance 

There  was  substantial  improvement  in  self-efficacy  for 
and  interest  in  being  adaptable,  even  after  accounting  for 
students'  individual  characteristics  associated  with  adaptive 
performance. 

Results  suggest  an  increase  in  the  perceived  need  for  adaptive 
performance  in  the  students'  current  jobs. 

Knowledge  about 
course  concepts 

Students  showed  increased  knowledge  of  AWALP  concepts. 

AWALP  was  successful  at  fostering  knowledge  gain  for  a  wide 
range  of  students. 

Team 

adaptive 

performance 

Students'  and  guides'  ratings  of  requirements  for  adaptive 
performance  generally  were  similar  to  each  other  across  the 
three  exercises.  Evidence  for  improved  student  accuracy  in 
ratings  of  requirements  over  time  was  mixed. 

Both  students  and  guides  gave  favorable  ratings  of  team 
performance,  but  students  seemed  to  become  more  confident 
about  the  level  of  team  adaptive  performance  as  the  course 
progressed. 

Application  of 
AWALP  principles 
on  the  job; 
attitudes  over 
time 

Students  reported  substantial  application  of  AWALP  principles, 
especially  in  coaching,  training,  delegating  to  subordinates, 
and  seeking  subordinate  input. 

While  AWALP  principles  were  successfully  disseminated  to 
graduates'  subordinates,  dissemination  was  somewhat  less 
successful  to  peers  and  commanders  and  throughout  the  unit. 

Graduates  remained  positive  about  AWALP  training  but  found 
command  climate  and  entrenched  leadership  the  biggest 
obstacles  to  applying  AWALP  principles. 

Graduates'  supervisors  were  positive  about  effect  of  AWALP 
on  graduates,  but  interpreting  results  requires  caution  because 
the  sample  size  was  small. 

course  to  others.  Very  few  students  noted  any  weaknesses  with  the 
course,  and  of  those  students  who  provided  recommendations,  most 
gave  suggestions  aimed  at  enhancing  a  course  that  they  already  felt 
was  effective.  In  addition,  interviews  of  course  graduates  showed  that 
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reactions  remained  favorable  over  time,  although  graduates  faced  chal¬ 
lenges  in  applying  the  principles  they  learned  in  AWALP  in  Army  units. 

Changes  in  Learners 

Changes  in  learners  were  evident  in  several  outcome  measures  of 
attitudes  and  knowledge.  Analysis  of  pretraining  and  posttraining 
survey  responses  showed  large  changes  in  self-efficacy,  indicating  that 
students  felt  more  capable  in  all  dimensions  of  adaptive  performance 
and  in  putting  adaptability  concepts  into  leader  practice.  These  find¬ 
ings  converge  with  results  of  prior  AWALP  course  evaluations,  in  which 
students  reported  being  more  prepared  to  be  adaptable  as  a  result  of  the 
course  (see  Chapter  Two),  but  the  current  results  are  less  subject  to 
demand  characteristics  and  do  not  show  ceiling  effects.  Likewise,  the 
current  evaluation  found  that  students  were  more  interested  in  being 
adaptable  in  their  jobs  and  in  encouraging  adaptability  in  the  teams 
they  lead.  Moreover,  improvements  in  self-efficacy  and  interest  were 
substantial  even  after  accounting  for  individual  characteristics  associ¬ 
ated  with  adaptive  performance.  Results  suggest  that  students’  percep¬ 
tions  of  their  current  jobs  also  changed,  as  they  recognized  a  greater 
need  for  adaptability  in  their  roles  with  respect  to  all  the  adaptability 
dimensions  and  most  of  the  leader  behaviors  assessed. 

Changes  in  learners  were  also  evident  in  terms  of  greater  knowl¬ 
edge  of  course  material.  Analysis  of  pretests  and  posttests  showed  sub¬ 
stantial  increase  in  declarative  knowledge  of  AWALP  concepts.  This 
improvement  was  independent  of  students’  general  cognitive  ability  or 
education. 

Evidence  of  learning  was  more  mixed  in  ratings  of  team  behavior. 
Students  were  generally  accurate  in  rating  the  requirements  for  adap¬ 
tive  performance  in  team  exercises,  in  which  accuracy  corresponded 
to  guides’  ratings.  The  accuracy  of  students’  judgments  of  adaptabil¬ 
ity  requirements  appeared  to  have  improved  over  time,  although  this 
result  could  also  reflect  decreasing  variability  in  guides’  ratings,  cor¬ 
responding  to  relatively  “flat”  student  ratings  across  the  exercises.  In 
rating  performance,  both  the  guides  and  the  students  agreed  that 
teams  performed  at  least  somewhat  effectively  across  most  adaptive 
performance  dimensions,  even  as  exercises  became  more  complex  and 
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demanding.  Consistent  ratings  of  performance  from  guides  on  exer¬ 
cises  with  increasing  levels  of  difficulty  could  indicate  behavioral  learn¬ 
ing.  However,  students  tended  to  inflate  ratings  of  team  effectiveness 
relative  to  guides’  ratings,  and  these  discrepancies  did  not  improve  over 
time  (and,  indeed,  were  most  extreme  at  the  end  of  the  course). 

Application  of  AWALP  Principles  on  the  Job 

Results  of  interviews  indicate  substantial  professional  change  in 
graduates  after  three  months,  especially  in  the  areas  of  coaching, 
training,  delegating  to  subordinates,  and  seeking  subordinate 
input.  Graduates’  responses  also  demonstrated  application  of  outcomes- 
based  strategies  on  the  job.  For  most  respondents,  positive  effects  of 
AWALP  were  sustained  six  months  after  graduation.  Reported  changes 
were  less  common  in  some  areas,  such  as  conducting  AARs  and  brief¬ 
ing  commanding  officers,  largely  because  graduates  reported  limited 
opportunities  to  engage  in  these  activities  in  their  current  roles. 

AWALP  principles  were  disseminated  most  successfully  to  gradu¬ 
ates’  subordinates.  Less  success  was  reported  in  disseminating  princi¬ 
ples  to  peers  and  commanders  and  throughout  the  unit.  These  findings 
appear  to  be  consistent  with  students’  expectations  about  challenges  in 
applying  AWALP  principles  on  the  job;  at  the  end  of  the  course,  stu¬ 
dents  most  often  anticipated  that  leader  buy-in  and  command  climate 
would  pose  obstacles  to  implementation.  Three  months  after  return¬ 
ing  to  their  jobs,  graduates  again  identified  leadership  and  climate  as 
the  principal  impediments.  Although  the  supervisors  we  talked  with 
were  positive  about  the  effect  of  AWALP  on  graduates’  behavior  after 
the  course,  results  should  be  interpreted  with  caution  because  of  the 
extremely  small  sample  size. 


Improving  AWALP 

Overall,  results  showed  only  limited  need  for  improvement  in  AWALP. 
The  following  subsections  summarize  and  discuss  possible  modifica¬ 
tions  to  the  course. 
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Address  Challenges  to  Implementing  Adaptive  Performance 
Concepts  on  the  Job 

On  the  last  day  of  AWALP,  students  create  training  plans  to  imple¬ 
ment  once  they  return  to  their  units.  This  activity  encourages  students 
to  consider  the  realities  of  applying  adaptability  concepts  in  their  jobs. 
As  shown  in  the  posttraining  surveys,  students  anticipated  that  leader 
buy-in  and  command  climate  would  be  key  challenges  to  implement¬ 
ing  adaptive  performance  principles,  and  indeed,  these  were  the  pre¬ 
dominant  challenges  graduates  reported  after  returning  to  their  jobs. 
Over  time,  we  expect  that  receptivity  to  adaptability  concepts  will 
improve  as  a  critical  mass  of  soldiers  participates  in  the  course  or  learns 
about  adaptability  in  other  Army  training.  In  the  meantime,  however, 
we  recommend  explicitly  discussing  in  AWALP  these  potential  obsta¬ 
cles  and  strategies  to  address  them.  After  AWALP,  creating  a  commu¬ 
nity  of  practice  among  graduates,  with  access  to  an  online  forum,  can 
provide  ongoing  support  by  enabling  leaders  to  share  experiences  and 
best  practices.  This  approach  has  been  successful  in  fostering  tactical 
knowledge  acquisition  among  Stryker  brigade  teams  (Hallmark  and 
Gayton,  2011). 

Provide  More  Feedback  to  Students 

Although  students  generally  were  satisfied  with  the  feedback  that 
guides  provided,  using  the  team  rating  instrument  (at  the  end  of  Chap¬ 
ter  Four)  within  an  AAR  could  further  enhance  feedback  and  might 
foster  convergence  of  students’  and  guides’  ratings,  which  remained 
discrepant  on  day  7  of  the  course,  and  could  contribute  to  enhanced 
team  interaction  and  performance  (Zaccaro  et  ah,  2009).  However, 
because  sharing  of  guides’  ratings  with  the  students  may  be  viewed 
as  inconsistent  with  a  constructivist  instructional  approach,  which 
emphasizes  learning  through  self-awareness,  the  students  could  be 
shown  only  their  own  team’s  ratings;  the  guides’  ratings  could  be  kept 
for  their  own  use  in  the  AAR  or  presented  only  after  the  AAR. 

There  are  several  ways  that  the  ratings  could  be  valuable,  even  if 
only  the  students’  ratings  are  used.  First,  students’  ratings  could  point 
to  topics  that  are  particularly  important  for  guides  to  probe  in  the  AAR 
(e.g.,  where  there  are  large  discrepancies  between  students’  ratings  and 
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the  guide’s  judgment),  and  the  form’s  structure  can  ensure  that  impor¬ 
tant  topics  are  not  missed  in  the  discussion.  Second,  displaying  the  aver¬ 
age  of  students’  ratings  for  each  team,  as  well  as  variation  within  teams, 
could  he  the  impetus  for  discussion  in  the  AAR.  For  example,  guides 
might  use  the  results  to  inquire  why  students  thought  they  did  particu¬ 
larly  well  or  poorly  in  particular  adaptive  performance  dimensions.  Or, 
considerable  disagreement  among  students  in  ratings  of  some  dimen¬ 
sions  (some  rate  the  team  high,  and  others  rate  it  low)  could  indicate 
that  the  team  does  not  have  a  shared  mental  model;  guides  therefore 
might  ask  the  students  to  discuss  their  positions  to  encourage  build¬ 
ing  a  common  frame  of  reference.  Third,  asking  students  to  complete 
the  team  rating  form  before  and  after  the  AAR  (rather  than  just  before) 
might  also  increase  convergence  of  subsequent  ratings  by  encouraging 
students  to  reflect  more  deeply  on  their  performance.  Reviewing  stu¬ 
dents’  pre-AAR  and  post-AAR  ratings  could  reveal  whether  students 
are  assimilating  performance  feedback. 

AWG  should  also  ensure  that  guides  use  the  team  rating  forms 
consistently.  Reliability  among  guides  can  be  assessed  by  having  mul¬ 
tiple  guides  independently  rate  the  same  team  during  practical  exer¬ 
cises  or  in  practice  sessions  using  observations  of  video  recordings  of 
the  teams. 

Guide  ratings  of  students’  individual  effectiveness  could  be 
another  source  of  feedback.  We  discuss  this  topic  in  the  section  on 
future  evaluation  of  AWALP  later  in  this  chapter. 

Update  and  Expand  on  Instruction  of  Team  Adaptability 

The  Pulakos  et  ah,  2000,  taxonomy  used  as  a  basis  for  AWALP  is  a 
model  of  individual  adaptive  performance.  However,  the  AWALP  cur¬ 
riculum  emphasizes  performance  in  teams.  While  some  of  the  Pula¬ 
kos  et  ah,  2000,  dimensions  of  adaptive  performance  may  be  relevant 
to  teams,  Baard,  Rench,  and  Kozlowski,  2014,  argues  that  there  may 
be  additional  dimensions  of  team  adaptive  performance  that  have  not 
yet  been  identified.  In  addition,  model  development  since  the  incep¬ 
tion  of  AWALP  has  advanced  the  theory  of  team  adaptation  and 
related  constructs.  One  notable  example  is  Burke,  Stagl,  et  ah,  2006, 
which  posits  an  input-throughput-output  model  of  team  adaptation  in 


Summary,  Conclusions,  and  Recommendations  89 


which  individuals  coming  together  as  a  team  develop  team  processes 
for  behavioral  results  (i.e.,  team  actions),  resulting  in  team  adaptation 
(i.e.,  team  innovation  or  modification),  as  reflected  hy  new  or  modified 
structures,  capacities,  or  actions.  Inputs  consist  of  individual  and  joh 
characteristics.  Throughputs  consist  of  (1)  four  phases  of  team  adapt¬ 
ability  (situation  assessment,  plan  formulation,  plan  execution,  and 
team  learning);  (2)  emergent  cognitive  and  attitudinal  states  (shared 
mental  models,  team  situation  awareness,  and  psychological  safety — 
the  shared  belief  that  the  team  is  safe  for  risk  taking);  and  (3)  cognitive 
processes  (e.g.,  cue  recognition,  including  recognition  of  the  need  to 
adapt)  and  interpersonal  processes  (mutual  monitoring,  communica¬ 
tion,  backup  behavior,  and  leadership),  through  which  the  four  phases 
are  enacted.  While  AWALP  already  addresses  most  of  the  factors  in  the 
Burke  et  al.  model,  the  curriculum  could  be  enhanced  by  reinforcing 
concepts,  including  planning,  which  entails  determining  a  course  of 
action,  setting  goals,  and  identifying  members’  roles  and  responsibili¬ 
ties;  mutual  monitoring,  whereby  team  members  observe  one  another 
in  an  attempt  to  catch  and  correct  errors  or  slips  in  a  timely  way;  and 
backup  behavior,  when  team  members  help  others  who  are  having  dif¬ 
ficulty  meeting  their  goals.  The  Burke  et  al.  model  may  also  be  useful 
in  AWALP  as  a  way  to  provide  instruction  about  the  relationships 
among  the  inputs,  throughputs  and  outcomes.' 

Enhance  Instruction  of  Adaptive  Performance  Dimensions 

The  evaluation  results  suggest  that  some  fine-tuning  of  course  content 
may  enhance  outcomes  for  a  few  of  the  adaptive  performance  dimen¬ 
sions.  For  example,  some  variation  in  responses  about  learning  to  handle 
ambiguous  situations  suggests  that  instruction  in  this  area  might  be 
strengthened.  Students  reported  significantly  greater  self-efficacy  for 
this  dimension  of  adaptive  performance;  however,  guides’  ratings  of 
performance  effectiveness  for  one  aspect  of  dealing  with  ambiguity  in 


*  Some  of  the  concepts  of  the  Burke  et  al.  model,  such  as  psychological  safety,  may  be  more 
applicable  to  teams  with  enduring  membership;  in  AWALP,  as  in  the  Army  more  generally, 
team  composition  changes  frequently.  Thus,  it  would  be  important  to  identify  potential 
boundary  conditions  of  the  Burke  et  al.  model  for  Army  teams,  the  impact  of  these  condi¬ 
tions,  and  strategies  to  overcome  their  possible  constraints  on  team  adaptability. 
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teams — adjusting  actions  rapidly  to  changes  in  the  situation — tended 
to  he  quite  a  hit  lower  than  students’  self-ratings.  In  addition,  guides’ 
ratings  of  team  effectiveness  in  generating  innovative  ideas — which  is 
one  aspect  of  thinking  creatively — suggest  that  this  aspect  of  the  course 
could  he  enhanced.  While  the  course  already  included  a  number  of 
practical  exercises  that  call  for  innovative  thinking,  it  could  he  useful 
to  incorporate  structured  brainstorming  techniques,  such  as  brainwrit¬ 
ing  (Paulus  and  Yang,  2000),  in  which  group  members  exchange  writ¬ 
ten  ideas  in  a  round-robin  fashion.  AWG  would  need  to  determine  the 
extent  to  which  these  areas  are  problematic  and  whether  they  can  be 
addressed  with  modihcations  to  the  course.  At  the  same  time,  adapt¬ 
ability,  while  malleable,  may  be  slow  to  change  (Ployhart  and  Bliese, 
2006).  Thus,  more  time  may  be  needed  for  students  to  practice  apply¬ 
ing  AWALP  principles  to  show  improvement  in  these  behaviors. 

Additional  reinforcement  of  cultural  adaptability  as  a  general 
construct  may  also  be  warranted.  Although  students  reported  high  self- 
efficacy  for,  and  interest  in,  cultural  adaptability,  as  well  as  greater  need 
for  cultural  adaptability  in  their  current  jobs,  the  absolute  level  of  need 
for  cultural  adaptability  was  lower  than  what  we  observed  for  other 
adaptive  performance  dimensions.  We  conjecture  that  some  students 
view  cultural  adaptability  as  more  about  national  culture  than  about 
different  social  groups  more  generally;  the  latter  is  the  focus  of  AWALP. 
This  supposition  is  partly  supported  by  a  lack  of  substantial  improve¬ 
ment  in  responses  to  one  of  the  questions  on  the  knowledge  test:  On 
the  pretest,  22  percent  of  the  students  incorrectly  answered  a  question 
about  cultural  adaptability,  indicating  that  it  pertains  only  to  a  situa¬ 
tion  involving  U.S.  Army  personnel  and  Afghans;  on  the  posttest,  18 
percent  of  the  students  selected  this  incorrect  answer. 

Reinforce  Knowledge  of  Adaptive  Performance  Concepts 

One  topic  for  AWG  to  consider  is  whether  students’  scores  on  the 
posttraining  test  are  acceptable.  As  reported  in  Ghapter  Three,  aver¬ 
age  scores  on  the  knowledge  test  were  60  percent  correct  pretraining 
and  76  percent  correct  posttraining.  If  the  posttraining  scores  are  not 
satisfactory,  additional  reinforcement  of  course  concepts  during  class¬ 
room  instruction  and  practical  exercises  may  be  needed  to  ensure  that 
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students  assimilate  the  material.  While  the  course  does  not  emphasize 
learning  definitions,  students  need  sufficient  mastery  of  course  con¬ 
cepts  to  teach  subordinates  about  adaptive  performance  and  to  con¬ 
vince  leaders  and  peers  that  change  is  necessary  within  units.  Like¬ 
wise,  if  ratings  of  team  performance  effectiveness  are  not  satisfactory, 
additional  strategies  may  be  needed  to  enable  students  to  demonstrate 
adaptability  in  team  contexts. 


Ongoing  and  Future  Evaluation  of  AWALP 

In  the  following  subsections,  we  recommend  ways  to  strengthen  ongo¬ 
ing  and  future  evaluation  of  AWALP.  To  provide  context  for  these  rec¬ 
ommendations,  we  begin  by  enumerating  both  the  strengths  and  limi¬ 
tations  of  the  current  study. 

Study  Strengths  and  Limitations 

The  study  had  a  number  of  strengths: 

•  We  used  multiple  methods,  measures,  and  types  of  respondents  to 
provide  a  comprehensive  evaluation. 

•  Measures  targeted  multiple  levels  of  adaptive  performance,  includ¬ 
ing  the  individual  and  the  team  and  the  intersection  between 
them. 

•  We  used  a  pretest-posttest  design  that  controlled  for  individual 
characteristics  associated  with  adaptive  performance.  This  allowed 
us  to  make  inferences  with  confidence  about  the  effect  of  AWALP 
on  training  outcomes. 

•  We  collected  data  from  students  several  months  after  graduation 
to  evaluate  application  of  course  principles  on  the  job. 

•  The  evaluation  demonstrates  methods  for  measuring  intangible 
training  outcomes  that  did  not  result  in  ceiling  effects  common 
in  studies  with  similar  self-report  measures  (Baard,  Rench,  and 
Kozlowski,  2014).  Thus,  the  approach  may  be  applicable  to  evalu¬ 
ating  a  wide  range  of  training  efforts  that  emphasizes  21st-cen¬ 
tury  soldier  skills. 
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However,  in  addition  to  these  strengths,  there  are  also  limitations: 

•  Participation  in  AWALP  is  not  random,  and  there  may  he  selec¬ 
tion  effects  in  terms  of  students  who  participate  (e.g.,  that  they 
are  selected  for  having  strong  leadership  skills  or  potential).  We 
attempted  to  control  for  these  effects  hy  measuring  individual 
characteristics.  Nonetheless,  generalizahility  of  the  results  to  a 
broader  sample  of  students  needs  to  he  tested. 

•  Because  this  was  a  field  study  rather  than  a  randomized  controlled 
trial,  it  is  not  possible  to  attribute  improvement  in  outcomes  or 
effects  on  participants’  jobs  unequivocally  to  the  course. 

•  We  relied  largely  on  survey  methods;  therefore,  our  outcomes 
may  still  be  subject  to  biases  or  other  problems  that  occur  with 
self-report  measures,  such  as  social  desirability  (as  suggested  by 
the  peer  ratings)  and  careless  responding.  On  the  other  hand,  as 
noted  in  Roselle,  2013,  and  as  evident  in  the  prior  surveys  used 
to  evaluate  AWALP  discussed  in  Chapter  Two,  the  changes  that 
we  observed  in  students’  ratings  (e.g.,  self-efficacy)  may  underesti¬ 
mate  the  effect  of  the  course  because  students  learn  that  they  were 
not  as  adaptable  as  they  thought  initially. 

•  Although  we  used  established  instruments  to  measure  some 
constructs,  we  created  original  items  for  others  (e.g.,  declarative 
knowledge  and  team  performance)  with  unknown  construct 
validity. 

•  Finally,  while  results  indicate  that  some  transfer  of  training  is 
occurring,  our  results  were  based  primarily  on  graduates’  percep¬ 
tions  of  their  behavior;  we  do  not  have  independent  assessments 
of  their  job  performance  because  of  challenges  in  recruiting  grad¬ 
uates’  supervisors  for  follow-up  interviews. 

We  next  propose  additional  approaches  to  measuring  outcomes  to 
address  some  of  the  limitations,  reduce  the  response  burden,  and  make 
the  evaluation  effort  more  efficient. 

Recommended  Measures  and  Processes  for  Ongoing  Evaluation 

The  following  subsections  summarize  recommendations  for  the  ongo¬ 
ing  evaluation  of  AWALP  and  provide  some  more  in-depth  discussion 
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on  each  point.  While  we  recommend  continuing  use  of  the  current 
instruments,  we  propose  changes  in  how  they  are  used. 

Continue  Use  of  Current  Instruments 

We  recommend  that  AWG  continue  to  administer  the  knowledge  test 
and  questions  about  attitudes  toward  adaptive  performance  and  leader 
practice  (self-efficacy,  interest,  and  need)  at  the  beginning  and  end  of 
training  and  to  administer  the  questions  measuring  reactions  to  the 
course  at  the  end  of  training.  In  addition  to  obtaining  feedback  about 
the  course,  continuing  to  collect  this  data  can  provide  the  volume  of 
responses  needed  for  more  comprehensive  assessment  of  reliability  and 
validity  of  the  measures.  We  propose  a  number  of  changes  in  use  of 
these  measures,  as  discussed  in  the  next  section. 

We  also  recommend  continuing  to  collect  information  from 
graduates  through  telephone  interviews  to  assess  how  they  are  using 
AWALP  principles  and  to  identify  lessons  learned  to  improve  AWALP. 
We  suggest  interviewing  all  graduates  in  the  next  two  to  three  AWALP 
classes,  then  sampling  students  from  subsequent  classes.  We  discuss 
collecting  data  from  supervisors  in  the  section  on  future  evaluation  of 
AWALP. 

Modify  Selected  Measures  in  the  Surveys  and  Team  Ratings 
Instrument 

We  propose  a  number  of  revisions  to  the  survey  instruments  to  fur¬ 
ther  reduce  response  burden.^  First,  for  routinely  evaluating  AWALP, 
we  recommend  eliminating  the  measures  of  personality  characteristics 
and  dispositional  traits  from  the  pretraining  survey,  although  AWG 
may  wish  to  include  demographic  characteristics  to  document  and 
understand  the  student  population(s).  However,  if  AWG  seeks  to  con¬ 
duct  additional  research  on  the  criterion-related  validity  of  AWALP 
training  (i.e.,  the  association  of  training  outcomes  with  subsequent  job 
performance),  it  would  be  useful  to  include  measures  of  cognitive  abil¬ 
ity  and  personality  and  dispositional  traits  (conscientiousness,  extra¬ 
version,  openness  to  experience,  and  learning  goal  orientation).  The 


^  As  indicated  in  Chapters  Three  and  Four,  we  recommend  eliminating  peer  evaluations 
and  “importance”  ratings  in  future  evaluations. 
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International  Personality  Item  Pool,  which  measures  the  Big  Five  per¬ 
sonality  traits  (Goldherg  et  ah,  2006),  and  the  Button,  Mathieu  and 
Zajac,  1996,  measure  of  learning  goal  orientation  are  available  in  the 
public  domain. 

Second,  our  team-level  measures  focused  on  how  adaptive  per¬ 
formance  dimensions  manifested  at  the  group  level,  but  AWALP  also 
focuses  on  other  aspects  of  team  collaboration  and  performance.  In 
concert  with  our  recommendation  to  modify  the  curriculum  to  address 
team  processes  in  more  depth,  we  recommend  modifying  the  team 
rating  instrument  to  include  assessment  of  group-level  process,  such 
as  planning,  mutual  monitoring,  and  backup  behavior  (Burke,  Stagl, 
et  ah,  2006,  and  Smith-Jentsch  et  ah,  1998;  see  also  Shanahan  et  ah, 
2007).  Such  an  instrument  as  the  Anti-Air  Teamwork  Observation 
Measure  (ATOM)  could  also  be  used.  The  ATOM  was  developed  as 
part  of  the  Tactical  Decision  Making  Under  Stress  project  and  mea¬ 
sures  behaviors  reflecting  four  dimensions  of  team  processes:  informa¬ 
tion  exchange,  communication,  supporting  behavior,  and  leadership 
and  initiative  (Smith-Jentsch  et  ah,  1998). 

We  provided  instruments  with  proposed  revisions  and  the  ATOM 
in  a  separate  report  to  AWG. 

Manage  and  Streamline  Data  Collection  Procedures 

Because  the  current  evaluation  was  conducted  for  research,  we  used 
some  data  collection  procedures  that  are  not  required  if  AWG  evaluates 
AWALP  for  quality  improvement  on  an  ongoing  basis.  Simplified  pro¬ 
cedures  can  ease  response  burden  and  may  improve  student  responsive¬ 
ness.  For  example,  we  protected  student  confidentiality  by  not  sharing 
individual  results  with  anyone  else  outside  the  research  team,  includ¬ 
ing  guides.  For  some  measures,  such  as  students’  ratings  of  their  teams, 
responses  were  submitted  anonymously.  Students  might  respond  more 
carefully  to  the  questions  if  they  were  aware  that  guides  would  review 
the  results  (e.g.,  on  the  knowledge  test)  or  display  them  in  AARs  (e.g., 
students’  ratings  of  teams);  at  the  same  time,  the  data  should  be  col¬ 
lected  in  a  way  that  encourages  students  to  be  forthright.  Therefore,  we 
recommend  not  sharing  individual  scores  or  ratings  (e.g.,  in  the  case 
of  ratings  of  teams,  guides  could  average  ratings  of  members  within  a 
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team).  When  possible,  ratings  should  be  collected  anonymously  (or  by 
team  number  rather  than  student  name),  but  some  responses  must  be 
identified  to  calculate  pretraining-posttraining  differences.  This  can  be 
done  by  assigning  code  numbers  to  students.  Whether  code  numbers 
or  names  are  used,  we  recommend  maintaining  the  confidentiality  of 
students’  test  scores  and  survey  responses. 

We  also  recommend  administering  the  instruments  online  rather 
than  using  paper-and-pencil  instruments.  Collecting  the  data  through 
computers  or  mobile  device  can  eliminate  errors  that  occur  through 
data  entry  of  paper  surveys,  and  responses  can  be  scored  immediately. 
Computerized  entry  can  also  provide  information  about  the  quality  of 
students’  responses  (i.e.,  “think  time”).  For  example,  low  scores  on  the 
knowledge  test  or  response  patterns  on  surveys  (e.g.,  selecting  the  same 
response  to  each  question)  coupled  with  extremely  short  response  times 
may  indicate  that  students  did  not  take  the  test  or  survey  seriously. 
(Long  response  times,  on  the  other  hand,  are  less  meaningful,  as  one 
does  not  know  what  respondents  are  doing  during  the  time  intervals.) 

Recommendations  for  Future  Evaluation  of  AWALP 

Additional  methods  and  measures  can  contribute  to  a  more  compre¬ 
hensive  evaluation  of  AWALP.  Our  recommendations  are  as  follows: 

•  Assess  training  transfer  and  results: 

-  Develop  forms  for  guides  to  rate  the  effectiveness  of  individual 
performance  in  AWALP. 

-  Analyze  association  of  effectiveness  in  AWALP  with  subsequent 
adaptive  performance  effectiveness  on  the  job  using  supervisor 
ratings  or  360-degree  feedback. 

-  Compare  results  for  trained  and  untrained  individuals  or  units. 

•  Conduct  behavioral  observations. 

•  Create  additional  indicators  of  AWALP  impact: 

-  Assess  sustainability  of  graduates’  attitudes  toward  AWALP 
and  toward  adaptability. 

-  Track  other  measures  of  AWALP  success,  such  as  repeat  busi¬ 
ness  and  requests  for  local  instantiations  of  adaptability  train¬ 
ing. 
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The  subsections  below  discuss  these  recommendations  in  more  detail. 
Many  of  these  approaches  are  also  relevant  to  other  courses  that  focus 
on  adaptability.  A  theme  of  many  of  these  recommendations  is  the 
need  to  obtain  additional  measures  of  adaptable  performance. 

Assess  Training  Transfer  and  Resuits 

The  most  important  topic  for  future  evaluation  of  AWALP  is  assessing 
transfer  of  training  by  assessing  its  criterion-related  validity.  A  study  of 
this  nature  would  assess  the  association  of  outcomes  measured  in  train¬ 
ing  and  individual  characteristics  with  subsequent  job  performance, 
where  graduates’  supervisors  would  provide  quantifiable  ratings  of 
job  performance  effectiveness.  (This  would  also  require  developing  a 
rating  form  for  supervisors  to  complete.)  Comparing  performance  rat¬ 
ings  for  graduates  with  a  matched  control  group  of  leaders  who  have 
not  attended  AWALP  would  strengthen  this  approach.  This  approach 
could  be  extended  to  studying  the  effectiveness  of  teams  whose  leaders 
or  members  have  (or  have  not)  participated  in  AWALP,  or  to  results  of 
entire  units  that  have  (or  have  not)  gone  through  adaptability  training.^ 
While  studies  of  criterion-related  validity  could  use  students’  self- 
efficacy  ratings  as  a  predictor  variable,  a  stronger  approach  would  use 
expert  (i.e.,  guide)  ratings  of  the  effectiveness  of  individual  performance 
in  AWALP  (e.g.,  see  section  on  behavioral  observations  below).  We  did 
not  ask  the  guides  to  rate  individual  students  because  the  study  already 
imposed  a  rather  large  response  burden  and  because  we  attempted  to 
obtain  ratings  of  individuals  from  peers  (which  was  not  successful). 
In  addition  to  providing  a  source  of  data  to  study  the  criterion-related 
validity  of  AWALP,  guide  ratings  of  individual  effectiveness  in  AWALP 


^  When  undertaking  a  study  of  training  transfer,  we  also  recommend  revisiting  the  research 
literature  on  the  association  of  individual  characteristics  and  adaptability.  Baard,  Rench, 
and  Kozlowski,  2014,  notes  the  need  for  stronger  theory  about  individual  differences  and 
adaptive  performance.  Other  recent  efforts  indicate  that  the  relationships  may  be  complex 
(Dorsey,  Cortina,  and  Luchman,  2010;  LePine  et  ah,  2000;  Zaccaro,  2007).  For  example, 
LePine  et  ah,  2000,  suggests  that  the  dependability  facet  of  conscientiousness  may  be  det¬ 
rimental  to  making  decisions  when  the  rules  change,  while  the  achievement  facet  of  con¬ 
scientiousness  may  be  associated  with  more-accurate  decisions.  Zaccaro,  2007,  argues  that 
combinations  of  traits  that  are  integrated  in  theoretically  meaningful  ways  are  more  likely  to 
predict  leadership  than  are  independent  contributions  of  single  traits. 
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could  serve  as  a  source  of  feedback  to  students  and  could  be  compared 
with  students’  knowledge  test  scores  and  self-efficacy  ratings  to  assess 
the  construct  validity  of  these  measures. 

For  the  criterion  measure,  i.e.,  supervisors’  ratings  of  graduates’ 
performance,  modifying  recruiting  processes  may  improve  success 
in  contacting  supervisors.  Recommendations  to  recruit  supervisors 
include  the  following: 

•  Track  who  sends  students  to  AWALP  to  obtain  supervisor  contact 
information  on  enrollment.^ 

•  Contact  immediate  supervisors  (for  platoon  sergeant,  contact  the 
platoon  leader;  for  section  leader,  contact  the  platoon  sergeant, 
etc.). 

•  Given  frequent  personnel  changes,  contact  supervisors  after  three 
months  rather  than  later. 

•  Give  priority  for  future  AWALP  enrollments  to  supervisors  who 
provide  feedback  about  course  graduates. 

Assessing  transfer  of  training  could  be  expanded  further  by  col¬ 
lecting  360-degree  feedback  for  graduates  and  a  matched  sample  of  lead¬ 
ers  who  have  not  attended  AWALP.  Using  360-degree  feedback,  rather 
than  relying  on  supervisory  ratings  alone,  might  reduce  common  rater 
errors,  such  as  halo  error  (see  Baard,  Rench,  and  Kozlowski,  2014).  The 
Army  already  has  a  360-degree  evaluation  process  in  place  for  officers 
in  the  Multi-Source  Assessment  and  Feedback  Program  (U.S.  Army, 
undated),  which  could  serve  as  a  starting  point  for  this  effort.  Many 
of  the  items  on  the  evaluation  form  are  directly  relevant  to  adaptabil¬ 
ity;  examples  include  “Adapts  quickly  to  new  situations  and  require¬ 
ments”;  “Improves  ability  in  interpersonal  interaction”;  “Maintains 
relevant  cultural  understanding”;  “Is  open  to  diverse  ideas  and  points 
of  view”;  and  “Goaches  others  in  the  development  or  improvement  of 
skills.”  However,  because  some  of  the  items  on  the  form  are  at  odds 
with  adaptability  principles,  the  form  would  need  some  revision.  For 


^  Tracking  enrollment  in  this  way  would  also  facilitate  tracking  repeat  business  (i.e.,  mul¬ 
tiple  enrollments  on  the  part  of  individual  supervisors)  to  use  as  an  additional  indicator  of 


AWALP  success. 
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example,  “Prioritizes  tasks  for  teams  or  groups”  and  “Makes  appropri¬ 
ate  assignments  or  role  delegation  to  subordinates  or  teams”  convey  a 
top-down  rather  than  a  bottom-up  approach.  A  related  effort  could 
examine  the  attitudes  and  knowledge  of  graduates’  (and  a  matched 
sample’s)  subordinates  to  assess  whether  graduates  have  successfully 
disseminated  AWALP  principles  to  the  soldiers  that  they  lead. 

As  described  by  Salas  et  ak,  2012,  a  wide  range  of  other  factors 
influence  training  transfer.  These  can  include  characteristics  of  trainees 
(additional  to  those  examined  in  this  study),  how  training  is  designed 
and  executed,  and  features  of  the  work  environment.  A  comprehensive 
analysis  will  examine  these  factors  to  understand  and  improve  training 
transfer. 

Conduct  Behavioral  Observations 

We  recommend  conducting  systematic  behavioral  observations  during 
the  course  to  evaluate  individual  and  team  performance  and  to  assess 
performance  improvement  as  the  course  progresses  (for  a  review  of 
behavioral  measures,  see  Wildman  et  ak,  2011).  We  propose  that  AWG 
use  event-based  measures  or  behavior  checklists,  in  which  trained 
observers  (i.e.,  guides)  record  critical  behaviors  as  determined  by 
subject-matter  experts.  In  AWALP,  critical  behaviors  would  be  linked 
to  the  dimensions  of  adaptive  performance  (e.g.,  see  White  et  ak,  2005) 
and  other  team  behaviors  relevant  to  each  exercise.  Event-based  mea¬ 
sures  have  been  used  in  prior  studies  of  team  adaptation  and  perfor¬ 
mance;  some  approaches  involve  documenting  the  frequency  of  specific 
behaviors,  while  others  include  more  detailed  assessments  of  the  qual¬ 
ity  of  those  behaviors  (e.g.,  see  Entin  et  ak,  1993;  Smith-Jentsch  et  ak, 
1998).  In  a  study  of  AWAEP,  Roselle,  2013,  used  behavior  checklists 
and  found  increased  frequencies  of  adaptive  behaviors  in  similar  practi¬ 
cal  exercises  conducted  at  the  beginning  and  end  of  the  course. 

A  primary  advantage  of  checklists  is  that  they  are  less  susceptible 
to  some  of  the  biases  common  in  self-report  measures  or  in  subjective 
ratings  of  the  frequency  or  quality  of  behaviors.  However,  checklists 
have  some  drawbacks.  They  are  more  labor  intensive  to  use  than  sub¬ 
jective  ratings,  and  an  effort  must  be  made  to  establish  interrater  reli¬ 
ability  among  observers.  Ratings  may  also  be  influenced  by  observers’ 
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expectations  about  performance  improvement  as  the  course  progresses. 
In  the  case  of  assessing  improvement  by  using  parallel  practical  exer¬ 
cises,  additional  challenges  include  establishing  the  equivalence  of  the 
exercises  and  requiring  added  time  in  the  curriculum.  Some  of  these 
issues  could  be  addressed  by  having  subject-matter  experts  who  are 
naiVe  to  the  course  structure  rate  video  clips  of  students  engaged  in 
course  activities.  This  approach  would  eliminate  the  need  for  equiv¬ 
alent  exercises  and  would  reduce  or  eliminate  biases  associated  with 
expectations  for  performance  that  might  occur  if  these  observations 
were  conducted  by  individuals  with  in-depth  knowledge  of  the  cur¬ 
riculum.  However,  because  these  ratings  would  be  conducted  post  hoc, 
they  would  be  more  useful  for  research  than  for  providing  feedback  to 
students  during  the  course. 

Create  Additional  Indicators  of  AWALP  Impact 

In  addition  to  measures  of  individual  effectiveness  training  and  sub¬ 
sequent  performance  on  the  job,  AWG  may  wish  to  assess  the  sus¬ 
tainability  of  graduates’  attitudes  toward  AWALP  and  toward  adap¬ 
tive  performance  principles  using  follow-up  online  surveys  beyond  the 
three-  or  six-month  points  after  graduation.  As  another  measure  of  suc¬ 
cess,  AWG  can  track  repeat  business  (i.e.,  multiple  enrollments  on  the 
part  of  the  same  units).  In  addition,  a  number  of  different  divisions 
have  asked  AWG  for  support  in  standing  up  local  adaptability  training 
modeled  after  AWALP.  These  requests  can  serve  as  other  indicators  of 
AWALP  success. 


Expanding  AWALP's  Approach  to  Adaptability  Training 

Below  we  discuss  two  possible  ways  that  AWALP’s  approach  to  adapt¬ 
ability  training  might  be  expanded.  One  option  is  to  increase  the 
number  of  students  receiving  training,  either  by  increasing  through¬ 
put  or  by  infusing  the  AWALP  approach  into  other  professional  mili¬ 
tary  education  (PME)  courses.  Another  option  is  to  create  a  follow-on 
course  that  expands  adaptability  instruction  at  the  team  level. 
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Increase  the  Number  of  Students  Receiving  Training 

The  Army  should  consider  ways  to  leverage  AWG’s  approach  and  dis¬ 
seminate  AWALP  principles  more  broadly.  We  propose  two  options 
for  expanding  AWALP.  One  is  to  increase  the  number  of  trainees  in 
the  existing  course  or  modified  versions  of  it.  AWG  could  continue  to 
stand  up  local  versions  of  AWALP  in  divisions  by  training  local  trainers 
or  through  mobile  training  teams. 

Another  way  to  disseminate  AWALP  more  broadly  is  to  incorpo¬ 
rate  adaptability  principles  into  existing  PME  courses.  Army  courses  for 
NGOs  that  currently  develop  adaptability  skills  include  the  advanced 
leader  and  the  senior  leader  courses.  For  officers,  relevant  Army  courses 
would  include  the  Basic  Officer  Leader  Gourse,  the  Gaptains’  Gareer 
Gourse,  and  intermediate-level  education.  To  modify  a  course,  the 
Army  would  first  have  to  determine  specifically  how  AWALP  training 
differs  from  the  course’s  current  approach  to  training  adaptability  skills 
and  then  determine  the  actions  needed  and  the  resources  required  to 
make  the  appropriate  changes.  For  example,  successfully  transform¬ 
ing  existing  courses  would  require  additional  training  development 
resources  to  modify  the  POI  and  course  materials,  along  with  instruc¬ 
tor  training  to  implement  the  new  approach.  It  might  also  require  dif¬ 
ferent  facilities,  equipment,  personnel,  or  other  training  resources  at 
training  bases.  In  addition,  evaluations  would  be  needed  to  assess  the 
degree  of  improvement  in  training  outcomes  and  determine  whether 
the  improvements  are  worth  the  costs  of  the  change. 

To  support  expansion  of  AWG’s  approach  to  adaptability  train¬ 
ing,  we  recommend  that  AWG  create  a  training  support  package  with 
a  POI  and  supplementary  materials  to  support  diffusion  of  adaptability 
training  to  the  Army  at  large.  While  a  conventional  Army  POI  (e.g., 
specifying  tasks,  conditions,  and  standards)  may  not  be  appropriate  for 
AWALP  in  light  of  its  teaching  approach,  a  set  of  organized  materials 
will  help  preserve  institutional  knowledge  related  to  training  adaptabil¬ 
ity  and  will  support  dissemination  of  course  content  and  instructional 
methods.  It  would  also  enable  assessment  of  how  well  guides  in  other 
implementations  of  adaptability  training  adhere  to  the  principles  of  the 
original  implementation.  Research  in  other  domains  (e.g.,  evidence- 
based  practice  of  medicine;  Watkins  et  ah,  2011)  suggests  that,  even 
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with  clear  and  strict  guidelines,  there  may  he  deviations  from  protocol; 
given  the  teaching  approach  applied  in  AWALP,  it  may  he  even  more 
important  to  exercise  care  and  attention  to  appropriate  implementation. 

Create  a  Follow-On  Course  That  Expands  Instruction  at  the  Team 
Level 

TRADOC  can  support  mission  command  principles  further  hy 
expanding  on  instruction  of  team  adaptive  performance,  as  described 
earlier.  AWALP  provides  a  starting  point  for  training  soldiers  to  work 
in  and  lead  teams,  hut  there  are  many  additional  topics  to  address  to 
foster  effective  teams  and  effective  team  leaders.  Some  of  these  topics 
(e.g.,  shared  mental  models)  could  he  addressed  in  more  depth  in 
AWALP,  hut  given  the  large  and  growing  literature  relevant  to  team 
adaptation,  we  recommend  developing  a  follow-on  course  focused 
exclusively  on  these  topics.  In  addition,  some  existing  leader  develop¬ 
ment  institutional  courses  or  other  training  that  focuses  on  making 
teams  more  effective  might  provide  an  appropriate  context  for  this 
training.  Examples  of  prospective  team-hased  topics  to  expand  upon 
or  to  address  in  training  include  the  following: 

•  Shared  mental  models.  The  concept  of  team  or  shared  mental 
models  is  discussed  in  AWALP  and  could  he  explored  in  more 
depth  in  other  courses  so  that  soldiers  understand  how  shared 
mental  models  influence  team  adaptation,  how  to  foster  shared 
mental  models,  and  the  conditions  in  which  team  mental  models 
are  heneficial  or  detrimental  to  performance.  Mental  models  are 
frameworks  of  knowledge  and  processes  that  represent  and  guide 
our  interpretation  of  data  in  the  world.  Team  mental  models, 
which  are  distinct  from  the  sum  of  team  members’  individual  cog¬ 
nitions  (Klimoski  and  Mohammed,  1994),  influence  the  process 
and  content  of  decisionmaking  (Walsh  and  Fahey,  1986).  Shared 
mental  models  are  important  for  team  training,  performance,  and 
adaptability,  can  reduce  intrateam  conflict,  and  influence  adap¬ 
tive  decisionmaking  in  a  variety  of  domains  (Cannon-Bowers, 
Salas,  and  Converse,  1993;  Fiore,  Ross,  and  Jentsch,  2012;  Lim 
and  Klein,  2006;  Marks,  Zaccaro,  and  Mathieu,  2000;  Randall, 
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Resick,  and  DeChurch,  2011).  Interestingly,  research  has  demon¬ 
strated  that  similar  mental  models  are  more  heneficial  in  novel 
situations  than  in  common  or  routine  situations  (Marks,  Zaccaro, 
and  Mathieu,  2000).  Unfortunately,  persistent  mental  models 
can  also  he  detrimental  to  adaptahility.  Case  studies  of  disasters 
have  shown  a  recurring  theme  of  poorly  constructed  and  obsti¬ 
nate  mental  models  in  the  face  of  changing  situations  (Smith  and 
Dowell,  2000;  Weick,  1990;  Weick,  1993).  In  particular,  dynamic 
conditions  require  teams  to  reinvent  their  models  (Uitdewilligen, 
Waller,  and  Zijlstra,  2010). 

•  Transactive  memory  systems.  These  are  related  to  shared  mental 
models  hut  are  specific  to  memhers’  knowledge  about  “who 
knows  what”  in  the  team.  Transactive  memory  systems  have  been 
observed  to  be  a  significant  predictor  of  team  performance,  group 
learning,  and  creativity  (e.g.,  Austin,  2003;  Gino  et  ah,  2010; 
Liang,  Moreland,  and  Argote,  1995)  and  are  particularly  benefi¬ 
cial  when  engaging  in  nonroutine  work  or  in  turbulent  conditions 
(Akgiin  et  ah,  2005;  Ren,  Carley,  and  Argote,  2006).  Students 
in  a  team-training  course  can  learn  how  to  foster  development 
of  these  systems,  e.g.,  by  composing  stable  teams  (Akgiin  et  ah, 
2005;  Littlepage,  Robison,  and  Reddington,  1997),  composing 
smaller  versus  larger  groups  (e.g.,  Jackson  and  Moreland,  2009), 
training  team  members  on  their  tasks  together  versus  individu¬ 
ally  (Liang,  Moreland,  and  Argote,  1995),  and  investing  in  the 
planning  stages  to  determine  who  has  what  expertise  (Rulke  and 
Rau,  2000). 

•  Team  trust.  Mutual  trust  is  an  underlying  principle  of  develop¬ 
ing  cohesive  teams,  which  in  turn,  is  a  central  element  of  mission 
command  doctrine  (ADP  6-0,  2012).  Trust  is  related  to  the  con¬ 
cept  of  psychological  safety  discussed  earlier  (Burke,  Stagl,  et  ah, 
2006).  A  team  training  course  can  address  how  trust  develops  in 
teams  and  how  to  foster  mutual  trust  and  cohesion. 

•  Process  losses.  When  working  on  cognitive  tasks,  such  as  gen¬ 
erating  ideas,  making  judgments,  and  solving  problems,  teams 
are  subject  to  a  variety  of  process  losses  (Steiner,  1972)  or  factors 
that  prevent  teams  from  being  as  effective  they  can  be.  Examples 
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of  common  process  losses  include  suppression  of  ideas,  failure  to 
share  information  that  is  critical  to  the  task,  polarized  decisions, 
and  pressures  toward  uniformity.  Numerous  studies  have  diag¬ 
nosed  the  causes  of  such  losses  and  have  identified  strategies  to 
avoid  these  pitfalls  of  team  interaction  (e.g.,  for  a  brief  review,  see 
Straus,  Parker,  and  Bruce,  2011). 

•  Team  facilitation.  Team  facilitation  can  foster  performance; 
for  example,  in  brainstorming,  skilled  facilitators  can  help  teams 
avoid  process  losses  by  keeping  the  group  focused  on  the  task, 
by  encouraging  contributions  without  criticism  (thereby  reducing 
evaluation  apprehension),  and  by  soliciting  contributions  from 
quieter  group  members  (thereby  reducing  social  loafing)  (Offner, 
Kramer,  and  Winter,  1996;  Paulus  et  ah,  2006).  Facilitators  can 
learn  to  use  a  variety  of  other  strategies  that  pertain  to  group 
composition,  structure,  and  process  to  enable  teams  to  capitalize 
on  members’  knowledge  and  skills  (see  Straus,  Parker,  and  Bruce, 
2011).  While  AWALP  provides  some  opportunities  for  students  to 
serve  in  team  leadership  roles,  additional  practical  experience  in 
using  strategies  for  facilitation  could  be  beneficial. 

Other  topics  in  research  on  teams  that  are  relevant  to  adaptabil¬ 
ity  include  goal  setting,  conflict  resolution,  and  working  in  distributed 
groups. 


Lessons  for  Adaptability  Evaluation  in  Other  Training 
Contexts 

This  research  can  provide  lessons  about  measuring  adaptability  and 
other  intangible  concepts  not  only  in  AWALP  but  in  other  training 
contexts.  In  addition  to  using  the  methods  and  measures  developed 
for  this  study  to  assess  local  versions  of  AWALP,  the  research  could  be 
applied  to  designing  evaluations  of  PME  courses  that  address  adap¬ 
tive  performance.  This  will  become  particularly  important  as  the  Army 
seeks  to  push  adaptability  training  into  the  mainstream  institutional 
training  domain. 
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The  evaluation  demonstrates  the  benefits  of  using  multiple  mea¬ 
sures  and  methods  and  documents  how  such  measures  can  be  devel¬ 
oped  and  implemented.  Evaluation  of  other  courses  that  involve 
adaptability  training  can  make  use  of  most  of  the  reaction  measures 
implemented  in  this  study  “as  is”  or  with  minor  modifications.  Pre¬ 
post  knowledge  tests,  attitudinal  measures  (e.g.,  self-efficacy),  and  post¬ 
graduate  interviews  can  be  used  with  appropriate  revisions  for  course 
content.  Instructor  and  student  ratings  of  team  performance  may  be 
broadly  applicable  to  Army  training.  Continued  use  of  these  instru¬ 
ments,  including  the  modifications  and  more  comprehensive  efforts 
as  suggested  above,  can  provide  data  to  validate  the  measures,  sup¬ 
porting  TRADOC  and  contributing  to  the  research  literature.  More 
fundamentally,  robust  studies  of  transfer  performance  could  provide 
evidence  that  adaptability  is  indeed  an  aspect  of  performance  that  can 
be  improved  through  training.  This  would  rebut  alternative  views,  such 
as  Individual  ADAPTability  theory  (e.g.,  Ployhart  and  Bliese,  2006), 
which  conceives  of  adaptability  primarily  as  a  set  of  individual  differ¬ 
ences  that  are  relatively  stable  and  thus  better  attained  in  an  organiza¬ 
tion  by  recruiting  the  right  kind  of  individual  rather  than  by  training. 

In  addition,  it  has  been  over  a  decade  since  Pulakos  and  col¬ 
leagues  developed  their  model  (Pulakos  et  ah,  2000),  and  the  body  of 
theory  and  empirical  research  on  adaptability  has  since  been  growing, 
particularly  at  individual  and  team  levels.  For  example,  adaptation  is 
defined  in  AWALP  as  a  performance  construct,  which  Baard,  Rench, 
and  Kozlowski,  2014,  classified  as  an  individual  difference.  Alternative 
theoretical  approaches  consider  adaptation  as  performance  change  or 
as  a  process,  which  are  more  conducive  to  training  (rather  than  selec¬ 
tion)  strategies  (Baard,  Rench,  and  Kozlowski,  2014).  The  course  mate¬ 
rial  in  AWALP  already  emphasizes  many  of  the  processes  by  which 
individuals  and  teams  can  be  adaptive  that  are  explicated  in  these  alter¬ 
native  perspectives,  and  use  of  one  of  these  theoretical  models  may  be 
helpful  as  an  organizing  framework  for  the  course.  AWG  should  also 
continue  to  stay  apprised  of  the  growing  literature  to  ensure  that  the 
key  concepts  addressed  in  the  curriculum  are  consistent  with  research 
on  adaptation. 
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Conclusion 

The  shift  in  Army  doctrine  from  command  and  control  to  mission  com¬ 
mand  calls  for  profound  changes  in  leader  and  team  conduct.  General 
Dempsey  has  noted  that  mission  command  principles — which  empha¬ 
size  a  bottom-up  rather  than  a  top-down  approach — must  become 
institutional  in  Army  doctrine  and  in  Army  training  (Dempsey,  2011). 
Thus,  this  change  in  philosophy  requires  a  concomitant  transforma¬ 
tion  in  training.  AWG’s  successful  development  and  implementation  of 
AWALP  exemplifies  mission  command  principles  in  terms  of  both  the 
content  of  the  course  and  how  it  is  taught.  AWALP,  supported  by  sys¬ 
tematic  course  evaluation,  provides  a  promising  approach  for  the  Army 
as  it  seeks  to  further  develop  adaptable  leaders  and  teams. 


APPENDIX  A 


Typical  AWALP-Like  Practical  Exercise:  One  Rope 
Bridge 


To  preserve  the  integrity  of  the  course,  the  following  example  does  not 
come  from  the  AWALP  curriculum,  hut  it  is  typical  of  AWALP  practi¬ 
cal  exercises. 


107 


108  Innovative  Leader  Development 


Purpose 


"One  Rope  Bridge"  is  an  exercise  that  illustrates  many  of  the  teaching  concepts  and 
the  instructional  approach  used  in  AWALP. 

In  this  exercise,  each  member  of  a  four-  to  five-person  team  is  taught  a  different 
knot;  collectively,  these  enable  the  group  to  construct  a  one-rope  bridge.  The  team 
must  figure  out  where  and  how  each  knot  should  be  located  to  construct  the  bridge 


Dimensions  of  Adaptive  Performance  and  Other  Skiils  Taught 

•  Learning  new  tasks  and  procedures  •  Critical  thinking 

•  Dealing  with  ambiguous  situations  •  Accountability 

•  Problem  solving  •  Teamwork 

Actions 


Students  watch  a  short  video  about  a  vehicle  losing  its  load  on  a  public  roadway. 
The  guide  leads  a  brief  discussion  about  the  video  and  presents  objectives  and 
expectations  for  the  activity. 

The  "pretest"  phase  of  the  exercise  is  used  to  determine  whether  any  participants 
are  familiar  with  the  knots  being  taught;  these  students  then  help  coach  their 
teammates.  Alternatively,  the  first  team  member  who  masters  the  knots  then  assists 
the  others. 

In  phase  1,  team  members  work  independently  to  learn  specific  knots.  Peer  training 
is  followed  by  practice  and  a  check  on  learning.  There  is  no  reference  to  a  rope 
bridge  in  this  phase. 

Students  are  given  a  break  prior  to  phase  2.  This  can  be  a  lunch  break  or  a  delay  as 
long  as  1-2  days. 

In  phase  2,  teams  are  presented  with  a  practical  problem  that  requires  all  members 
of  the  team  and  their  equipment  to  cross  an  obstacle  in  20  minutes.  Essentially,  the 
task  requires  the  team  to  construct  a  one-rope  bridge.  Collectively  the  group  will 
know  how  to  tie  the  component  parts  of  the  bridge;  however,  they  must  solve  the 
problems  of  where  and  how  to  place  each  knot  to  construct  the  bridge. 


AAR  Questions 


Phase  1 

How  did  we  go  about  helping  you 
develop  a  new  skill? 

How  difficult  was  this  for  you? 
Were  you  focused  on  passing  the 
test  or  getting  better? 

Did  time  cause  you  any  concerns? 
How  does  this  model  compare  to 
how  the  Army  traditionally  teaches 
a  class? 


Phase  2 

What  was  key  to  the  group's 
success? 

Did  you  accomplish  the  mission? 

If  not,  what  was  the  reason(s)? 

Did  you  complete  the  mission? 
What  was  the  learning  environ¬ 
ment  like  (both  phases)? 

How  was  the  learning  activity 
structured? 

Characterize  the  instruction 
What  21st-century  skills  were  exer¬ 
cised  here? 


APPENDIX  B 


Items  Assessing  Reactions  to  Training 


Below  are  the  questions  used  to  assess  reactions  to  training.  Items  with 
the  same  letter  in  parentheses  were  combined  into  scales  as  described 
in  Chapter  Three. 

Table  B.1 

Questions  Assessing  Reactions  to  Training 


Question 


Response  Options 


1.  AWALP  guides  were  knowledgeable  about  the  subject  Strongly  disagree  to 

matter  (a)  strongly  agree 

2.  AWALP  guides  effectively  facilitated  AARs  and  group 
discussions  (a) 

3.  AWALP  guides  effectively  facilitated  course  exercises 

(a) 

4.  AWALP  guides  provided  sufficient  feedback  on  my 
individual  performance 

5.  AWALP  guides  provided  sufficient  feedback  on  team 
performance 

6.  The  feedback  I  received  from  AWALP  guides  enhanced 
my  learning® 

7.  Team  members  provided  relevant  feedback  on  my 
performance 

8.  Course  materials  supported  the  learning  objectives 

9.  AWALP  training  facilities  were  satisfactory 
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Table  B.1 — Continued 


Question 


Response  Options 


10.  Please  indicate  the  degree  to  which  you  agree  or 
disagree  that  your  abilities  in  the  following  areas  have 
improved  as  a  result  of  participating  in  AWALP: 

a.  Performing  in  uncertain  situations  (b) 

b.  Training  others  (b) 

c.  Conducting  AARs  (b) 

d.  Interacting  with  people  from  different  cultures  (b) 

e.  Thinking  creatively  (b) 

f.  Being  self  aware  (b) 

g.  Solving  problems  (b) 

h.  Handling  emergencies  (b) 

i.  Performing  under  stress  (b) 

j.  Coaching  others  (b) 

k.  Being  confident  on  the  job  (b) 

11.  Pick  the  two  topics  from  Question  10a  -  k  above  that  Open-ended 

you  think  have  changed  most  as  a  result  of  AWALP. 

How  will  your  performance  in  these  two  areas  be 
different  in  the  future? 

12.  Pick  the  two  topics  from  Question  10a  -  k  above  that 
you  think  have  changed  least  during  AWALP.  Why  will 
your  performance  in  these  two  areas  not  be  different? 


13.  Overall,  1  was  satisfied  with  what  1  learned  from 

AWALP  (c) 

14.  Attending  AWALP  was  a  good  use  of  my  time  (c) 

15.  AWALP  is  relevant  to  my  career  (c) 

16.  1  would  recommend  AWALP  to  others  (c) 

Strongly  disagree  to 
strongly  agree 

a.  Why  or  why  not? 

Open-ended 

17.  The  length  of  the  course  was 

Too  short,  about 
right,  too  long 

18.  The  amount  of  time  spent  in  the  classroom  (compared 

Too  little,  about  right. 

to  hands-on  activities)  was 

too  much 

19.  The  classroom  portions  of  the  course  were 

Too  academic,  about 
right,  mostly  common 
sense 
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Question 

Response  Options 

20.  The  exercises  in  the  course  were 

Too  difficult,  about 
right,  too  easy 

21.  What  aspects  of  AWALP  should  be  changed?  How 
would  you  change  them? 

Open-ended 

22.  Are  there  any  other  ways  that  AWALP  will  change  the 
way  you  lead  others  that  you  haven't  described  in  your 
other  answers?  If  so,  how? 

23.  Do  you  anticipate  challenges  applying  what  you  have 
learned  in  AWALP?  If  so,  what  are  they? 

^  The  mean  rating  for  this  item  was  the  same  as  the  mean  for  Item  5  and  was  not 
shown  in  Figure  3.1. 


APPENDIX  C 


Main  Effects  and  Interactions  of  Individual 
Characteristics  and  Attitudes  Toward  Adaptive 
Performance 


Table  C.l  indicates  statistically  significant  main  effects  of  individual 
characteristics  on  self-efficacy  for  and  interest  in  adaptive  performance 
dimensions  and  leader  practice.  In  most  of  these  cases,  individual  char¬ 
acteristics  were  associated  with  pretraining  but  not  posttraining  out¬ 
come  measures. 

Table  C.2  shows  significant  interactions  of  individual  characteris¬ 
tics  and  time,  which  occurred  for  a  small  number  of  outcomes: 

•  The  change  in  self-efficacy  for  creative  thinking  depended  on 
openness  to  experience,  such  that  the  change  was  larger  for  stu¬ 
dents  who  reported  lower  openness  to  experience.  The  course  thus 
had  a  greater  impact  on  self-efficacy  for  creative  thinking  for  stu¬ 
dents  with  lower  levels  of  openness  to  experience,  and  the  course 
had  less  (or  no)  impact  on  self-efficacy  for  creative  thinking  for 
students  with  higher  levels  of  openness  to  experience,  holding  the 
other  individual  characteristics  constant.  The  change  in  interest 
in  creative  thinking  depended  on  motivation  for  training,  such 
that  students  who  anticipated  that  they  would  benefit  more  from 
the  course  showed  a  greater  increase  in  interest  for  creative  think¬ 
ing  compared  with  students  who  expected  fewer  benefits  from  the 
course. 

•  The  change  in  self-efficacy  for  interpersonal  adaptability  and  deci¬ 
sionmaking  under  stress  depended  on  time  in  service.  The  course 
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Table  C.1 

Main  Effects  of  Individual  Characteristics  on  Self-Efficacy  (E)  and  Interest  (I) 


Dimension 

Effect  or 

Characteristic 

Creative 

Thinking 

Ambiguous 

Situations 

Learning 

Interpersonal 

Cultural 

Decisionmaking 
Under  Stress 

Leader 

Practice 

Pre-post  change 

E,l 

E,l 

E,l 

E,l 

E,l 

E,l 

E,l 

Openness 

E,l 

Extraversion 

E 

E,l 

E 

E 

E 

1 

E 

Conscientiousness 

1 

Mastery  orientation 

E,l 

E,l 

E,l 

1 

E,l 

1 

1 

Performance 

orientation 

|3 

Time  in  service 

E 

E 

E 

E 

E 

Motivation  for 
training 

^  This  relationship  was  negative,  i.e.,  students  with  a  higher  performance  orientation  were  less  interested  in  being  in  situations 
requiring  decisionmaking  under  stress. 
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Table  C.2 

Interactions  of  Time  (Pre-Post  Scores)  with  Individual  Characteristics  for  Self-Efficacy  (E)  and  Interest  (I) 


Dimension 

Characteristic 

Creative 

Thinking 

Ambiguous 

Situations  Learning 

Interpersonai 

Cultural 

Decisionmaking 
Under  Stress 

Leader 

Practice 

Openness 

E 

Extraversion 

E 

Conscientiousness 

1 

Mastery  orientation 

E,l 

Performance 

orientation 

Time  in  service 

E 

E 

Motivation  for  training 

1 
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had  a  greater  impact  on  these  dimensions  for  students  with  less 
time  in  service,  and  the  course  had  minimal  or  no  impact  on 
these  dimensions  for  students  with  greater  time  in  service,  hold¬ 
ing  the  other  individual  characteristics  constant. 

•  The  change  in  self-efficacy  and  interest  for  cultural  adaptability 
depended  on  a  mastery  learning  goal  orientation.  The  course  had 
a  greater  impact  on  self-efficacy  and  interest  for  cultural  adapt¬ 
ability  for  students  with  lower  mastery  orientation,  and  the  course 
had  minimal  or  no  impact  on  self-efficacy  and  interest  for  cultural 
adaptability  for  students  with  higher  mastery  orientation,  holding 
the  other  individual  characteristics  constant.  The  change  in  inter¬ 
est  in  cultural  adaptability  also  depended  on  conscientiousness, 
showing  a  similar  pattern  of  results  (i.e.,  greater  change  in  interest 
in  cultural  adaptability  for  students  with  lower  conscientiousness). 


APPENDIX  D 

Interview  Questions 


Questions  for  Graduates 

1.  Are  there  ways  in  which  you  feel  you  have  changed  profession¬ 
ally  as  a  result  of  the  AWALP  training? 

2.  (If  respondent  does  not  address  these  topics  in  his  or  her  response): 
Have  you  changed  how  you  do  the  following?  Frequency  that 
you  do  the  following? 

a.  Coach  or  mentor  subordinates? 

h.  Train  others  in  technical  skills/procedures? 

c.  Train  others  for  physical  strength/endurance? 

d.  Train  others  to  work  effectively  in  teams? 

e.  Train  others  to  solve  complex  problems? 

f.  Conduct  an  AAR? 

g.  Brief  a  commanding  ofhcer? 

h.  Delegate  responsibilities  to  others? 

i.  Seek  and  use  input  from  subordinates  about  a  complex  deci¬ 
sion? 

3.  (For  behaviors  listed  above  that  did  not  change),  why  not? 
Possible  prompts: 

a.  Opportunity 

b.  Command  climate 

c.  Necessity  on  the  job 

d.  Value  of  AWALP  approach  over  my  previous  method 
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4.  (Prompts  if  interviewee  answers  “yes”  to  2a,  2b,  2c,  2f,  2g,  2h 
and  not  already  addressed  in  their  answer) 

a.  What  are  the  key  skills  and/or  personal  characteristics  nec¬ 
essary  to  effectively  (2a,  2b,  2c,  2f,  2g,  2h)? 

b.  In  what  ways  does  your  (2a,  2b,  2c,  2f,  2g,  2h)  differ  from 
before  AWALP? 

c.  Were  others  receptive  or  unreceptive  to  your  changes  to  (2a, 
2b,  2c,  2f,  2g,  2h)? 

i.  If  unreceptive,  how  did  you  handle  that? 

ii.  If  receptive,  how  did  they  respond? 

5.  (Prompts  if  interviewee  answers  “yes”  to  2d,  2e,  2i) 

a.  Describe  an  example 

ii.  What  was  the  situation? 

iii.  What  did  you  do? 

iv.  What  was  the  result? 

6.  (If  the  answers  to  2a-2i  were  “yes”):  We  are  interested  in  how 
your  experience  at  AWALP  affects  soldiers  in  your  unit/organi¬ 
zation. 

a.  Who  in  your  organization  has  been  affected  because  of  your 
changes? 

b.  Who  is  most  affected  by  those  changes? 

c.  Do  you  think  your  commander  is  aware  of  the  changes? 

d.  Who  is  the  best  person/people  at  the  unit  level  to  comment 
on  how  AWALP  has  affected  former  students? 

7.  Looking  back  on  your  experience  in  AWALP,  are  there  any 
changes  that  you  would  recommend  making  to  the  course? 

8.  Would  you  recommend  AWALP  to  others? 

a.  If  so,  what  type  of  person  could  benefit  the  most  by  the 
experience? 

b.  If  not,  why  not? 

9.  Do  you  have  any  other  feedback  about  the  AWALP  training  that 
you  did  not  share  through  the  end-of-course  survey  or  through 
this  interview? 
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Questions  for  Graduates'  Supervisors 

1.  Did  you  have  a  role  in  determining  who  went  to  AWALP? 

a.  If  yes,  what  were  the  reasons  for  selecting  whom  you  selected? 
Would  you  use  different  reasons  for  future  selections? 

2.  Have  you  noticed  ways  that  the  AWALP  graduates  changed  fol¬ 
lowing  their  return? 

a.  If  so,  how?  Possible  prompts: 

ii.  Interact  with  subordinates  (delegate,  train,  coach) 

iii.  Conduct  AARs 

iv.  Communicate  with  you 

3.  Do  you  have  any  other  comments  about  AWALP  training? 


APPENDIX  E 


Change  in  Students'  Pre-Post  Responses 
Regarding  Need  for  Adaptive  Performance 


These  figures  show  the  degree  of  change  in  students’  pre-post  response 
to  questions  about  the  need  for  adaptive  performance  in  their  current 
jobs.  The  response  options  were  “never,”  “a  few  times/year,”  “monthly,” 
“weekly,”  and  “daily.” 

For  example,  for  cultural  adaptability  shown  in  Figure  E.l, 
20  percent  of  the  students  changed  their  responses  by  two  or  more  cat¬ 
egories,  indicating  a  greater  need  for  being  adaptive  (e.g.,  from  “never” 
to  “monthly”  or  from  “monthly”  to  “daily”).  Thirty-two  percent  of  the 
students  changed  their  responses  by  one  category  (e.g.,  from  “monthly” 
to  “weekly”  or  from  “weekly”  to  “daily”).  Thirty-seven  percent  did  not 
change  their  answers.  Some  students  also  reported  less  need  for  being 
adaptive;  8  percent  dropped  by  one  category  and  3  percent  dropped  by 
two  or  more  categories. 
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Figure  E.1 

Degree  of  Pretraining-Postraining  Change  in  Responses  Regarding  Need 
for  Adaptive  Performance  Dimensions  in  Current  Job 


H  +2  or  more  H  +1  CZI  0  CZI  -1  CZI  -2  or  more 

RAND  RR504-E.1 
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Figure  E.2 

Degree  of  Pretraining-Postraining  Change  in  Responses  Regarding  Need 
for  Leader  Behaviors  in  Current  Job 


RAND  RR504-E.2 
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The  Asymmetric  Warfare  Group  offers  the  Asymmetric  Warfare  Adaptive  Leader 
Program  (AWALP)— a  10-day  course  designed  to  enhance  adaptive  performance 
in  leaders  and  promote  innovative  solutions  in  training  in  support  of  unified  land 
operations.  This  report  describes  results  of  a  systematic  evaluation  of  AWALP, 
offers  recommendations  to  improve  the  course,  and  provides  recommendations  for 
ongoing  evaluation  of  AWALP  and  other  courses  or  events  that  address  adoptive 
performance  and  acquisition  of  other  intangible  skills.  The  study  used  o  pretest- 
posttest  design  and  collected  data  from  104  students  who  participated  in  AWALP. 
Results  show  substantial  improvement  in  training  outcomes,  including  students'  self- 
efficacy  for  being  adoptive  and  leading  adoptive  teams  and  knowledge  of  course 
concepts.  Graduates  also  reported  that  they  were  applying  course  concepts  on  the 
job  after  returning  to  their  units.  In  addition,  students  hod  exceptionally  favorable 
reactions  to  AWALP  and  remained  extremely  positive  about  the  course  three 
months  after  graduation.  Results  indicate  few  needs  for  improvement  in  the  course; 
the  most  important  area  to  address  is  challenges  in  applying  concepts  on  the  job 
because  of  the  command  climate  and  entrenched  leadership.  Recommendations 
for  ongoing  evaluation  focus  on  obtaining  additional  measures  of  adoptive 
performance,  particularly  to  establish  the  impact  of  AWALP  on  subsequent  job 
performance.  The  current  success  of  AWALP  suggests  that  its  approach  to  training 
might  be  usefully  expanded  in  the  Army,  and  we  discuss  strategies  to  achieve 
broader  dissemination.  Finally,  we  describe  how  the  methods  used  in  this  study 
might  be  applied  to  evaluating  related  training  in  other  contexts. 
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